If he can't be trusted, surely it is better he stays in the house with the children when he sees them - or have I got the wrong end of the stick?
He is not entitled to come and go as he pleases, make sure you set proper times for him to visit.
You may think you have a crap deal, but he is paying the mortgage on a house he can't live in.
Your other alternative is to move out. As you have children the council will class you as a priority. Depending where you live it could be uncomfortable for a while.