It could be classed as racially insulting because the reference to African implies an attempt to use skin colour (the only visual that one could associate with the African continent) as if it were an insult.
Granted being called African ought not offend but having given it some thought, I think reference to skin colour, whether overt or implied, can be considered more aggressive. One doesn't usually consider, for example, skinny and fat people to be in competing rival groups; but skin colour implies different people, different tribes, a "them and us" situation.
Thing to do is refuse to play that game, not interpret it like that. Let it be water off a duck's back.