LittleStar12, simple answer, leave it be!! Years ago we had a cat that did the same thing, found out later it belonged to a lady up the street.
Now I have 3 cats of my own and was informed by a young couple who live 2 blocks away, that one of our cats visit them at the same time everyday. Only he goes into their house and sleeps on the back of their sofa, - if you please! They have also bought him a bed, so that he has a 'good place to sleep' during the day!!
Many cats opperate between 2 homes, but they will move on as quickly as they came. As long as the cat is not doing any damage to your garden, leave it be.