It's mainly confirmation bias, in that on occasion you will feel that someone is looking and you and they are, and you go "oh wow!", but there are just as many if not more times when you think someone is looking at you when no-one is.
This is not the same as saying "it doesn't happen" -- rather, there is little evidence to suggest that it's anything other than random chance, plus our usual inability to be any good at sorting out correlation from causation. Well, probably.