What you have to bear in mind is that any threat of legal action probably won't even faze them- they probably will be finished soon (If not as soon as they say) and therefore would think that any legal action would be a bluff; it would be far too lengthy and they would probably deem the matter too trivial for you to proceed with (I'm not saying this matter is a triviality).
That said, try this:
You are on my property, and under the principle as established by Entick v Carrington (Every invasion of land, be it ever so minute, is a trespass), you are a trespasser. If you fail to vacate the area stated, and restore my rightful access to the property, I will commence proceedings under the principle as expounded upon through Kelson v Imperial Tobacco, seeking damages and if necessary, an interlocutory injunction. Your co-operation in this matter would be greatly appreciated; to resolve this situation without legal representation would be of benefit to us both.
Failing that, take the pole out yourself and disavow all knowledge if the thing comes crashing down.
Oh, if I've misunderstood the fact about where the scaffolding is and it isn't actually inside your property, use the case of the 'Duke of Rutland'; it was established that one does not have to be on your property to trespass, the fact that the defendant was interfering with the enjoyment of land was enough.