From the Shelter website:
"Your landlord or agent is only entitled to keep all or part of your deposit if they can show that they have lost out financially because of your actions, for example, if you have caused damage to the property or you owe rent".
Source:
http://england.shelte...it_protection_schemes
As I can't see that you've lost out financially, I can see no valid reason for withholding any part of the deposit.
Chris