You say allowed to see, is there a reason for him not to see them, social services and the courts might think it inadvisable if there are circumstances that would put your children at risk.
If you have custody then it is your choice.
I work with children in these circumstances and I would get the father to make a commitment to them. Start by letters, phone calls and then a visit, if it works out. It should be taken at the kids speed and if it isn't good for them, then don't pursue it.
The kids will be confused as this will be a person that they don't know.
Hope this helps.