I don't think you are paranoid. I believe you probably have mild anxiety of social situations. Of course this affects all sorts of people, no matter what they look like, as it is a mental thing. Based purely on how you feel inside.
Just tell yourself if people are looking at you, they are not judging you, they may even admire you. Breath deeply, lift your head up and just walk.
How much attention do you pay to other people you see? Probably not much, so just remember noone is paying too much attention to you either.
Do you live in a city?