i expect if this person felt you were actually at least trying to stop your kids annoying them they may not feel so aggrieved.
we were very careful as we knew they would tell our parents and our parents would punish us
one of them was very spiteful though, i always remember him telling my dad he had seen me pick all the moss out from between the bricks in his wall and leave the bits on the floor - i hadnt at all and i remember his face looking at me as my dad slapped the back of my legs for it. he had obviously done it himself as there were no other children in the street. small road, 9 houses, and a dead end.
my dad hated this man and a knew he was just a bitter nasty weirdo but he slapped my legs to kind of placate him i think - to make him satified.
now im not suggesting you smack your kids legs etc - but if you were to show that you are willing to instill in your kids the need to shpw more consideration to them then they may be 'placated'
i am sure in general your kids are not bad kids - but kids by their nature are often pretty careless and thoughtless and dont realise that their actions can be upsetting ...
perhaps you should go to see this person, with the kids, and calmly talk about the issue - dont go in all guns blazing, accusing and trying to defend it - just accept that whatver the rights and wrong and leagalities of it, this person is upset and you would rather they werent if you dont want this to drag on
maybe the kids hearing all this and realiing the real trouble, may make then try a bit harder not to let the ball 'roll' into the garden etc
you could even ask the person if theyd be happy for your kids to put some sort of barrier when they play - buy a cheap kids goal and put it across their gateway to catch the ball. bot sure how thatd work, but might be worth considering it - as i say, showing willing to help them may just be all they wanted...