I had exactly the same problem with my ex (although I wasn't married and didn't have children). If you go to the police station and speak to them they can go and talk to him about harrassment. If you want to take it a step further you can go to court and get a non-molestation order out against him (this is what I did). My ex broke it once and I went straight down to the police station, we ended back up in court and he was told he would face punishment if he did it once more, he hasn't contacted me since (touch wood). Don't let him get you down, I know its hard but its probably what he wants. Good luck x