Where do the boundaries get drawn between having a laugh / a natter and excluding or picking out a person for group attention? There's no definitive answer. But PP, you are describing work bullies. There is a vast amount of this behaviour that goes unchecked or is excused as 'friendship groups', 'having a laugh / a natter'. And usually the target of the bullying is dubbed 'oversensitive', 'not a team player'.
It is however workplace bullying.
Sadly, in the absence of whoever is meant to be in charge realising that they need to sort it, there is nothing you can do. Bullies do not change. They might shift their focus or go quiet for a bit, but they will do it repeatedly.
People who fall in with bullies usually do so because they fear being the next target.
So typically, a female work group of this kind will have members who literally run to the gatherings so as not to be 'left out' and who fear not being there when the group is 'in session'.
Both men and women bullies use socialising outside of work to strengthen the gang - making sure X or Y know that 'everyone else' met up at theirs and had a brilliant time.
What advice can I give you, PP? Be true to yourself - you are very good to befriend the person who has been got at. If nothing is to be done make medium term plans to move desk / floor / department / job if you can. But I think you are doing the right thing by keeping out of their coven.