I'm with Bednobs on this one.
If you knew in December that the drop off time by the father was 4.00, why on earth did you book swimming lessons knowing that your child needs to be there at 3.45 in order to get ready for the lesson?
I do agree that the father sounds like he is being a bit awkward and I think he should be a little more flexible, but the times were agreed in court, and the father has every right to stick to them.
To not let the father pick up the child at all, as stupidly suggested by littlemissxx, is both unfair on the child and father, and would be in breach of a court order.