I think you're lumping together too many things though.
"tolerance/ understanding/ empathy".
I might not have a full understanding of what it's like to struggle with what I'd regard as "basic" maths -- I've never struggled with it. But then that's entirely different from being tolerant. I spent half the time helping out at said High School explaining the difference between radius and diameter again and again and again without losing patience (well, not outwardly). Last week I was teaching someone twice my age at least, going step-by-step through a simple problem. If anything he was the one getting irritated at his own "stupidity". I kept having to reassure him that it was anything but, that he just lacked the experience to see how to tackle the problem.
Oh, and for that matter, it's not like I don't have empathy either. I spent basically three months this year on a calculation that, if I understood what was going on a bit more, should have taken less than a day -- and the error was a minus sign in the wrong place, or accidentally multiplying something by two, and the like. I struggle with maths too, plenty of times. More than most people here, I'd reckon, although then it's my day job so I have to struggle with it.
And through all this, I love to teach because I genuinely love the subject and I genuinely love trying to share that passion with other people, and maybe help them enjoy maths too.
In which case it's a bit frustrating, once again, to have a conversation that runs along the lines of "oh so what do you do?" "Physics" "ah you must be an arrogant git then"...