Until a couple of hundred years ago, the area was, apparently, known as Wormholt Scrubs. Now - whilst 'holt' certainly means 'wood' or 'copse' - I can find no evidence that the plant 'wormwood' was ever called 'wormholt'.
Also, the original meaning of 'worm' was 'snake' or 'serpent'. It could, therefore, still easily be that the name originated historically in the idea of a scrubby woodland area infested with snakes rather than one where wormwood grew, just as the link I offered earlier suggested.
But what the hey!