I suspect that most people feel 'out of their comfort zone' with situations which they can't personally relate easily to. For example, someone who's been brought up in a wealthy, upper-middle-class home might not be prejudiced in any way against those who've had fewer advantages in life but still feel 'on edge' when surrounded solely by the type of people who feature in 'The Only Way Is Essex'.
Equally, a celebrity from TOWIE who doesn't resent the 'upper classes' might still feel 'edgy' when solely in the company of people with a public school background.
Similarly, a man who isn't sexually attracted to other men might feel 'out of his comfort zone' when he witnesses overt displays of such attractions, even though he isn't in any way homophobic.