I doubt that you've got any cause for complaint against your neighbours (or their visitors).
When I lived in a terraced house, I had a right of way across the gardens of two neighbours, to get to my back door. It was the convention in that area (on the northern outskirts of Sheffield) that nobody ever used their front doors. If you visited anyone along that road, you'd find that their front door was blocked by furniture or even sealed up. Everyone always walked across neighbours' back gardens to get to their own property. (If anyone saw you actually using your own front door, you'd have been regarded as extremely 'odd'). Everyone had their letterbox in their back door and milkmen always delivered to the rear of properties. In about thirteen years at that address, I never saw anyone in that area ever use their front door.
If you live in a similar area (or even if you don't but your house has similar covenants) you should expect everyone living at the neighbouring property, and everyone visiting it, to walk across your garden.
You've got reason to complain about your neighbours. Your only complaint, if any, should be against the solicitor who acted for you when you bought the house, for failing to explain the significance of the covenants relating to access.
Chris