There are a few on here who know their stuff on this sort of thing, and you'll have to hope they see it sometime.
My first thought though is engage a solicitor and seek legal advice. Some of the things they have done (remove plants) is not on.
I personally would have thought that some form of rights have been established, and just because they say one thing doesn't make it right.
The only thing I can say is that it is unlikely, having put up a fence, that they are going to claim some on your side is theirs as well, and you should be within your rights to do what you like on your side. That said, the fence is their property, so do not fix anything to it at all,