I'm not surprised you didn't say anything because well u don't expect someone to say that to you in cadburys, I would have been a bit 'what did he just say?'
But yes like the others say best not give him any credit by commenting back I would pretend I didn't hear him that way his joke/comment would have just fallen flat ..thats even more embarassing to him.
I find out more and more these days that when these loud mouth people come out the wood work and try and create 'a scene' in public I want to say something back and really come back even harder than them to make them back off, but at the end of the day these people seem to have very lttle to lose, they don't mind at all out physiclal fight, being arrested, or jail. And as I have a professional 'legal' job, no criminal record and assets to my name I'm the only one whos going to lose out. I used to be scared of confrontation growing up but mentally I have been through some bad times over the past couple of years that has really hardened me to a 'taking no **** from no-one' frame of mind. But at the same time I think I have to weigh up the risks of how far a disagreement might go purely because I well lose the most because they have nothing!