i too have been through court proceedings with my ex as he was taking my son to places that were not appropriate, and i stopped contact. as a mother you have the right to know where your children are and have contact numbers for your ex. i would wait for cafcas to contact you and tell them what has been going on, and that he willnot provide contact details. the law is very much on the mothers side so get a good solicitor and tell the judge that he does not acknowledge the childrens birthdays ex, and that the children do not want to go. believe me if you stand your ground you will win. but also have you tried to find out why the children don`t want to go as this also could be used in court to stop hi, getting contact. he sounds like he doesn`t deserve the right to see his children. does he pay child support coz if he doesn`t make his day and report him to the csa, taking money off the ex is so much sweeter when they think theyv`e got away with it.
don`t give up it will get better just stand your ground, gather evidence. good luck