I am no expert in the legal aspects of parental access, but if he asks for access, simply refuse, even if the court sides with him.
Go back to court, and take the medai with you - shame him into leaving you alone.
It may be a fight, but it has to be worth it.
I am so sorry to hear what a slimeball your husband has turned out to be - no woman - or child - has to put up with his from anyone - a valuable lesson your chldren are never to young - and your husband never too old - to learn.