I had to re-read this a couple of times, but I now think your employer is correct.
There are two ways to look at this, but they amount to the same result.
1. If you consider yourself a full-time worker, (you work a full 40 hour week like everyone else, but spread over 4 days) so you get the full holiday entitlement, rather than a pro rata'd (part time) entitlement.
However, each of your accrued holiday days represents a 'normal' 8 hour day, so if you take a day off, using one of your holiday days, then you actually still owe them the extra two hours you would normally work. that day. Over a four day week, you would thus owe 4 x 2 hours - ie a whole 8 hour day.
So to take your 4-day, 40 hour week off, you would need to use 5 x 8 hr 'holiday days'. (ie. as per factor30's answer)
2. The second way is to consider yourself a part-time worker. You work 4 out of 5 days, so get 4/5 (four fifths) of the holiday entitlement. If we say it's 20 days, then you would get 4/5 x 20 = 16 holiday days. However, each of these pro rata holiday days amounts to your normal working day (ie. 10 hours).
In this case, a week off would cost you 4 x 10 hour holiday days, as Friday is not your normal working day.
If I was an employer, I would use this second, pro rata method, because by giving you the full holiday entitlement, you could actually take much more leave than you were entitled to if you used up your allowance by taking a day off here and a couple of days there.