He'd need to sleep and attend to other bodily functions. He'll probably also spend a bit of time dodging the guards -- and perhaps trying to read the number over their shoulders when they open the safe to put even more money in.
Then, what if he forgets where he's got to?
On the up-side, with practice he'd probably get each try well below 15 seconds -- though why does a 3-number set take the same time as a 4-number one.
However, would he not be able to try each 3-number permutation on the way to the 4-number ones? In which case you could ignore the time needed for the whole 3-number set.
After a while, he might start wondering if there were actually five numbers, or if the number had been changed that time back in 1972 when he was on holiday...
Or the lock might wear out altogether.
Come on, this isn't really a very practical question, now is it...?