I agree with Eddie, if you possibly can keep this amicable. It's worked well for 4 years, so strive to keep it friendly. Put yourself in your ex's position, he's been used to having his son 50% of the time and that has worked well, now you suddenly say to him that that will in fact halve. He's going to be angry, upset and feel threatened, I'm sure you would if he had done the same thing to you, so just try to sit down and have a talk about it and find a reasoned way through. My ex and I have an amicable relationship which culminated in her moving abroad with my children, and I felt worried and slightly threatened briefly, but good communicaiton kept us on the straight and narrow which is better for the kids and better for both of you. I still see the kids as whenever I wish, they stay with me, and I go over there and stay with them. It takes effort but it's really worth it not to fall out with your child's other parent.