No, it's not a police matter but, because many people don't understand the law, the police are forced to investigate many complaints about things being 'offensive', even though they're actually perfectly lawful.
It is, thankfully, NOT an offence simply to cause offence to someone else (or to another group of people, such as those of a specific race, religion or sexuality). The display of an (allegedly) offensive banner only falls foul of the Public Order Act 1986 if it is done
(a) with intent to provoke violence ; OR
(b) with intent to cause fear of immediate unlawful violence ; OR
(c) with intent to cause (and thereby actually causing) a person harassment, alarm or distress ; OR
(d) where the content of the banner is threatening to a person likely to be caused harassment, alarm or distress ; OR
(e) where there is intent to stir up racial hatred or where, through the display of the banner, racial hatred is likely to be stirred up.
None of the foregoing provisions apply to to the display of the banner by Crystal Palace fans but, clearly, someone who doesn't understand the law has taken offence and wrongly thinks that 'offensive' is synonymous with 'illegal'. It ain't!!!