I won't give you advice, just tell you about my friend, her son, and his father.....
I watched this similar scenario happen to my friend's son when he was young and like you, she put up with it for the sake of the child.
Well - he's 21 now, and has a very good relationship with his father. He remembers being let down, and he remembers the tears and panic. But he doesn't let it get in the way of the relationship he now has with his father. His father was a fool, a young selfish fool - he didn't know how to relate to the child, but since the child eventually becomes a young adult (about 12, these days!!) he was able to develop a different, more adult relationship. They are both happy, and the young man has a great mum who brought him up very well adjusted indeed. Good luck to all of you!