There's a saying - a dog will give you lifelong love and affection, a cat will take a message and get back to you. Cats are just not as loyal as dogs, their loyalty is certainly not in proportion to the love you give them, as you have discovered. Your cat may simply be a nomad, fond of a change of scene, and may simply prefer the food and atmosphere at someone else's house, before moseying back home to see if things have changed or not. There's not a lot you can do about that - just love him the way you always have, and hope that he stays with you. Good luck.