Yep, agreed. For all your manager knows, you could have another job for those two days that you may actually be working at that week, so you can't be made to use holiday entitlement to cover days when you wouldn't be there anyway.
I have a similar contract, and if I want a week off (22 hours over three days), then I book leave only for the Tuesday, Wednesday and Friday that I would be working.
However, on my leave sheet, I still have to put my last working day as the Friday before and my first day back as the Tuesday the following week, but I still only have to book 3 days/22 hours (we have to put them both) on my leave sheet. I am only 'charged' three days' leave.