I think £20 per hour for a self-employed profession is fairly reasonable.
However, you charge the £20 per hour, to cover times when you don't/can't work due to weather, illness, holidays, etc ... so he's having a laugh when he's asking you for holiday pay!!!!
I'm a window cleaner and am self employed. I can just imagine what my customers would say to me if I told them I was going on holiday and still expected to be paid.
Make hay whilst the sun shines .... and put enough away for the times when you can't work - that's my motto.
A self employed person does NOT charge holiday pay - tell him to take a running jump.
Oh .... in reply to your original question, you don't need to give him any notice as there's no contract of any kind. As already suggested, tell him a family member is going to be doing it from now on.