How old is your son now and is the father named on his birth certificate?
My guess would be that, although he has a right to ask you if he can have your son over night etc, he cannot insist on it and should really accept what you say.
He may be wanting more time with him and if he staying somewhere thats four hours away, this may be his reason for wanting him to stay over. If your son is at an age where he can say yes I want to go stay with my dad.......you really should consider your sons wishes. If he is still a baby.........then the choice is totally yours I would think.........and to be honest, if I was in your shoes, I wuldnt let my baby go overnight with people he really doesnt know enough.