this is definately emotional abuse to the children MORE than you. you are right and if you want to stop it you have to be firm and get it set in stone ie court order that you want his visits supervised etc sounds mad but a social worker can help with this also by giving a report etc. how do your children behave when they are back with you. are they unhappy, bad tempered or treat you differently than before?
the reason im saying this is because my brother does not see his son now as his ex brainwashed the poor lad from the age of 2 to tell my brother that he didnt want to see him. my sister also has a little boy who is now nearly 7 and has been pushed from pillar to post emotionally, she now has a boy who to be quite honest i cannot stand being around as he is a horror, his father did the same that your ex is doing and he now thinks the only way to get attention and love is to kick off, he has been diagnosed as mild adhd which is twaddle its what he has had to deal with at a young age. my sister is stupid and thought things would just magically get better and now its too late. the child still says things like ' my dad loves me cos he gives me everything i want and you dont' she is constantly in tears and to be honest i have no sympathy for her as fair enough she stopped the access but still lets her ex's dad take him out who is far worse, this is because she has lost the will to try and deal with him.
im not saying you are anything like my sister as they definately broke the mold when she was made but im advising you do something to protect your childrens innocence and childhood so its not scarred with memories of this.
it does affect children so much more than people think. please get help with this as you have the power to stop him ruining your childrens lives and future relationships in adulthood.
i really hope it all works out for you x