I'm afraid I can't have too much sympathy for your ex husband's feelings as he apparently wanted to dump all the responsibility for bringing up his son during the difficult years but somehow thinks he's going to miraculously pick up the relationship again when your son is 18 and no longer a financial responsibility. Let your son make up his own mind over the years how he feels about this lack of care and contact. Your ex husband may find he's living in a dream world
If Phil really wants to keep in touch with your son, that's fine, and it will be good for him to have some male influence in his life, but perhaps gradually wean the contact down so that your son doesn't feel too much dependence on the relationship, or grieve for the loss too greatly if your boyfriend suddenly finds a new female on whom he wants to concentrate.
I think one of the key issues here, long term, for your son, might be the number of future boyfriends or "uncles" who might appear or disappear from your life as he is growing up, and the possible instability and feeling of inpermanance this might cause him. This will involve you thinking very seriously about the suitability of future male friends in the way they impact on your son. You are not a bad mother. You obviously care very deeply about your son's emotional welfare. Perhaps you just need a period of being a single mum for a while without the complications of a specific boyfriend until you've got your new home sorted and your emotional life back on a steady keel again. Do you have any male relatives - father, brother, uncles who might be a more permanent feature of your lives who can help provide this kind of male friendship for your son?