If you do not want to cause offence revoke your invitation.
The number of things from which Muslims can take offence are so many and varied (and very often totally illogical) that it is best to steer clear of the minefield entirely.
Your quandry demonstrates perfectly why "multiculturalism" is a nonsense. Non Muslims cannot possibly be expected to know of all the foibles from which Muslims suffer. Since, in the UK, giving offence to somebody runs a very close second only to murder in the list of the most heinous crimes imaginable, best to give such a possibility a wide berth.