If you can live with it, don't have a dog that could jump up at it (depending how high it is) or children then you could leave it.
However I had one quite low down in a flimsy shrub & I have a dog who was very interested in it & so sadly I had it removed. The man who came said that it could have become much, much bigger....apparently the wasps go on adding to it all of the time....& the sheer weight can bring them down from weak branches. If they burst open you've got big trouble!
I'm sure that's probably rare, but just giving you the 'worst case scenario' so you can make up your mind.