I assume that you mean a router, rather than a modem (although a router actually has a modem in it anyway).
The telephone system periodically tests your connection in order to provide you with the best speed. (It can take up a fortnight from first installing a router for the system to optimise itself completely). If the system can't detect your router it regards it as a fault on the system and then reduces your download speed to see if that helps matters. When you turn your router back on you'll only get the reduced speed until such time as everything is fully optimised again.
So turning your router off at night is definitely NOT to be recommended, as you'll permanently get lower download speeds than you could otherwise be achieving.