Height means nothing at all. If you care about each other, why bother what other people might say. They will move on to something else after a while.
I am short, my Husband is tall, and no matter if it is two men, two women, who cares. Get on with your life and enjoy it.
Good luck.