I don't think it needs populism or anything else to drive the polarisation. It is an inevitable consequence of the two viewpoints. (What is populism anyway but actually listening to the democratic will of the people and representing their views as a representative should ?)
One side wishes to remain being dictated to, to a large degree, by an external power block, of whom we have no say in who is appointed as the elites but will rule by right (of something). Seemingly not concerned about their individual national identity and concerned that having to stand up for ourselves will mean a transitional period of upheaval, slightly lower growth in the short to medium term and one of those spikes in the economy that happens from time to time and which over the longer term isn't a big deal.
The other are horrified that our proud and glorious nation may no longer exist as an independent unit, no longer controlled from within by it's elected representatives but near destroyed as an sovereign nation as it is subsumed by the mix of diverse cultures and foreign views that is the EU, and who hold sway and dictate to us what laws we shall have whether we like it or not. It inevitably seems a betrayal of the past, the present, and most of all the future generations.
So it polarises by it's very nature and affects many aspects of life and many issues we need to confront. And there will always be a few folk unstable enough to be unable to control their thoughts and actions when confronted with this sort of situation. The answer was to ensure the unstable were identified and not left on their tod, with their inability to look after their problem by themselves, out in society.