Do you not think that maybe encouraging him in a less negative way would actually make your son less apprehensive about the visits? At 6 he is probably totally out of the understanding depth needed to grasp the animosity between you and his dad, try not to influence him so much and let him go along with his dad with more of a positive and excited attitude, he'll need his dad all his life and if he ever realises that you had caused him to wrongfully hate his dad, he will then turn on you.
Blunt answer even for me but it is my opinion.