just a query - but if you've never stopped him seeing his son, why did he take you to court in 2005? i have to say though, i do think its a bit unfair when people allow other men in their lives to be called Dad by their children - it appears your son would have been 3 when your husband came into your lives (about the same time as your ex was taking you to court) - only you or your husband could have put that idea into your son's head.
I don't think you can ever take away PR from a parent though, even if he does seem a bit remiss in his role. does he live far away? you mentioned that he will not travel here for a day - how far is it? x