I think it is a huge insult to be called racist. On the few occasions I have been accused of it - unjustly, I may add - I have countered it with 'That's racist. Would you say that to me if I were black (or Asian, Polish, etc)?'
I get very very angry about it.