Charging from a USB port and charging from a wall socket will both fully charge your device, but due to the different currents, it will take a lot longer to fully charge using a USB connection.
Is the device fully charged when you decide to use it, or do you just assume it's fully charged due to how long it's been connected??
I have a similar problem with my Tom Tom. I rarely use it and when I do, I simply plug it in to get a little charge to get me going; then plug it in to my car which maintains the charge. It never gets fully charged though and you can guarantee that the next time I use it, I'll need to boost the charge again.
Try charging it using a plug/wall socket and make sure it is fully charged.
If it keeps losing charge, then there may be a fault with the battery itself.
https://www.techwalla.com/articles/usb-charging-vs-wall-charger