Nicola Jane,
While I can't begin to understand how this must affect you, as I have no experience, I would like to offer some comments. People can be most ignorant, and your reaction, whatever it may be, is unlikely to change that. The person who could be most affected by this is your son, and I think his reaction is probably the best example of how to deal with them. If he is genuinely unaware, then they are not hurting him, and by meeting their rudeness with a cheery response he is rising above it, and maybe, just maybe, one or two of them will be shamed by this response. The temptation to verbally abuse them must be strong, but by doing so you probably reinforce their hostility. However if it makes you feel better then I say - just you go ahead and tell them what you think!