I don't think the memory usage is a big deal: On my machine, IE8 uses 134,544 and Chrome uses 210,568. Firefox comes about about the same as your figure.
However, as I don't use FF much, I was on 7, so I've just upgraded to 8. It crashed during the upgrade, then it opened a welcome page for every addon, plus FF itself.
Then when I closed it, it remained running but not visible and I had to terminate it manually.
Finally, it constantly hangs when clicking a link in AB.
These are all the kinds of things that persuaded me to switch to Chrome.