You say that you don't give a hoot, and yet clearly you do if it makes you feel so uncomfortable when transpeople are out in public -- in one particular setting, perhaps. But then you throw out the "lifestyle" thing, and in itself that reveals rather a lot. It's hardly a lifestyle. It's how they are; there is very little choice involved.
That's why the "Pandering" thing is so sad to read. It shouldn't be seen as pandering to ask that you consider that gender isn't about body parts. It shouldn't be seen as pandering to ask that you allow people to go to the toilet *they* feel most comfortable in. After all, that's the same choice you have. It isn't pandering to ask to be able to live in public as who you are, when in any meaningful sense beyond perception this has no impact at all.
So yes, everything else is a distraction, because it is trying, consciously or otherwise, to hide the fact that people arguing against this are usually arguing against the "lifestyle", or at any rate don't understand it nearly as well as they think they do. The toilet thing is just another mainfestation of this -- a more concrete way to show that there is still a lack of acceptance.
How would you know, anyway? Some transpeople do kind of stick out rather; others don't, except perhaps on close scrutiny. Are you planning to institute some kind of vigorous checking process to determine what sex people are? Ironically, that would be even more of an invasion of privacy than the one you seem to be worried about. How about whether there should be a line between "pre-op" and "post-op" transpeople? And so on and so forth.
I would have thought, then, that the obvious "common sense" solution is to avoid all of this and either come out and say that transpeople aren't welcome in public toilets full stop, or that they are, and then deal with such abuses as they happen.