This is not easy because above all you really will want to do this without hacking off the neighbour too much. Equally it is never surprising how many people take the P.
What they have a private right of way across your land. Whether it is described sufficiently as a specific route (that can't easily be changed) you will have to discover by inspecting your Land Registry title plan and register. You can get copies of these here for �6.
http://www.landregisteronline.gov.uk/
Look at the two documents - the plan may describe a specific route - often hatched in a different colour on the plan - the register will describe in words the nature of the access and cross-refer to the plan.
Armed with that information, you can decide if you can change the route. It is perfectly feasible to 'fence off' or otherwise partition your garden to force the following of a specific route through it. Indeed owners of land on which public pathpaths run often do this and essentially 'give up' ownership of the land forming the public path by putting a 2 metre fence up alongside the public footpath to stop the public straying and to maintain their security.