Milk is zero rated for VAT. Reading the HMRC website, it says,
"Postage and delivery charges
HMRC Reference:Notice 700/24 (April 2003)
2.3 What if I have to deliver the goods and I make a charge for this?
The important test is whether delivery is included in the contract. You are making a single supply of delivered goods if, under the contract, you have to deliver the goods to a place specified by the customer. This might include the customer’s:
own address;
friends or relatives; or
their own customers.
The position is not affected by whether the charge you make for delivery is separately itemised or invoiced to the customer. In either case there is a single supply for which the VAT liability is based on the liability of the goods being delivered. >>>>>For example, any element of the price attributed to the doorstep delivery of milk and newspapers will also be zero rated. <<<<< On the other hand any element attributed to the delivery of standard rated mail order goods will be standard rated"
The way that reads to me, as the VAT-free milk is being delivered as part of the contract, no VAT can be charged on the delivery. You may want to query him with regards to the VAT he has charged you until now.