Im sorry, but if you were scared of him, then why have you left them with him? Why were you scared?
I fail to see why, scared of your ex or not, you have not seen them for 2 years
As bedknobs has said, you cannot just pick up after 2 years of not seeing them
If you love them so much you can't cope without them imagine how they must have felt when you walked out and left them with a father their own mother was scared of!
What you can do is start to send them letters. Call them on the 'phone even if they refuse to speak too you, call them. Have you been sending BD and Xmas cards/presents?
Even if they choose not to see you at the moment there will come a time when they will want some answers as to why you left - take the responsibility for it; you chose to leave without them and for that you must say sorry.