Agreed, we're not all "racists", only some of us. I, for example, am certainly a "racist" in the modern sense of the word (and in its original sense, too, but only if you substitute the word "culture" for "race" - I don't think race (=genes) has much to do with anything, on the other hand I certainly believe there are superior and inferior cultures).
I disagree with you, Khandro, that the term is used to end an argument which is being lost; the term is typically deployed either to stop a proper argument starting, or to constraint the way it may develop.
For example, a "racist" may ask why most gang rapists are Pakistani, or why most of the perpetrators and most of the victims of London's knife crimes don't have traditional English names. And he does this because he sees a correlation which he thinks needs accounting for: we have data which may be relevant - or may not - but, either way, we won't know until we ask the question and examine the evidence.
The non-racist disallows these questions because they carry the implication, if only hypothetically, that racial (I prefer "cultural") differences may account for differences in criminality. This possibility is excluded a priori by the non-racist's world view, and the "racist" is morally wrong (for all the obvious reason) for suggesting it.