Hi astro Sounds like a terrible situation.
If you dont let them go you will be in violation of the order - perhaps some legal eagle will know the consequences.
Sounds like you need to see a solicitor on Monday. Were you not represented at the original judgement? When you say the ex is not allowed near the kids........is that a court judgement? If he is living there that sounds like a good enough reason not to let them go - you need some specialist advice and quick.
What about offering to host a visit for the grandparents on Tuesday, so they come to your home?