I'm not an expert, and can just relate what happened to me a few years ago.
In my back garden have an apple tree which was growing at an angle. To try and rectify this, I tied string to the tree, and tied that to the adjoining fence, which, technically, was a neighbours. It wasn't a permanent fixture, and kept coming - so I thought - untied. Well to cut a long story short, it was the neighbour untying it all the time, and he did not want me touching his fence in any way. There was a row, I called him a few names, and in a fit of temper I kicked and damaged the fence. He was a horrible bloke, and not my first run in with him.
Anyway, he called the police, and I ended up getting a caution AND having to replace HIS fence panel.
Though different circumstances, it's not that different to your own in that it's attached to YOUR property. He didn't want me attaching anything to his property, so I would have thought that you would have the right not to have it attached to yours if you so wish. Don't hold me to that - I'm just relating my experience.