you have two jobs. If they were in different places, then one place would pay you what ever you are entitled to be paid when that place is closed. Its not paid on the condition that you don't work, its paid as part of your pay entitlement.
You could go off and work at the other place and they would pay you. Provided you are paying the appropriate tax and NI, there's no problem IMO. If they don't pay you to do the decorating, they will have to pay someone else. I think the hook is because its called "holiday pay" if it was called your "summer retainer" or your "ecklefecken payment" or anything else this issue wouldn't arise. Ask the payroll person where it is written that you can't do this.
Its not uncommon in the NHS for people to do agency or temp work while they are on leave, the only time it can't be done is if the person is taking mat leave.