It is not in any way illegal to take photos of anyone, adult or child, in a public place.
http://phooto.co.uk/rights.shtml
Owners of property do not have the right to prevent the photographing of their property from a public place.
It is illegal to harass another, and the taking of photographs could amount to harassment - if the 'victim' suffers alarm or distress and there has been at least two occurrences.
The UK does not recognise 'invasion of privacy' as such, but the European Convention on Human Rights gives everyone the right to respect for his private and family life, his home and his correspondence.
Using a telephoto lens into a person's home would probably be a contravention of this, regardless of where the photo was taken from. But taking photos of something any passing person can see probably doesn't. Would one expect an open front drive to be private? Is your drive clearly visible to the highway?
Sadly, it is not black and white.
However, telephone the police for advice. Tell the police this man's behaviour is frightening your children. Keep a diary of events - this will help your case.