Perhaps because they are perceived to be scruffy? Also I guess arguments could result from supporters of other teams starting on them so they are minimising risk.
Because they identify you as a supporter of that particular club which supporters of that teams rivals may take offence at and single you out for verbal/physical abuse