City Hall: A correct (even if somewhat over-zealous) response. The area around City Hall is indeed private property (as, for example, many shopping centres are) and the owners of that property are within their rights to allow, prohibit or partially-restrict photography as they deem fit.
Houses of Parliament: A correct response. You were in a public place so (with only limited exceptions) you're free to photograph what you want and in the way that you want to (unless you cause an obstruction).
Horseguards: Utter b0ll0x. You don't need a permit.
Trafalgar Square: There are bye-laws in place which restrict professional photography (and the wardens are extremely quick to move in on anyone who seems to be, for example, setting up publicity shots for a new band or product). However the use of a tripod does not constitute professional photography.
A recent report of Westminster CIty Council's Citizens and Business Scrutiny Committee recently specifically recommended that, in its parks and open spaces, "the use of photographic equipment should not be restricted, save that fees should be paid by professional photographers . . . ":
http://www.westminste...rt_WEB-1276597709.pdf
All of the other places you've mentioned appear to be public places so, as long as you don't cause an obstruction, there should be no restrictions on your photography. The only exception might be that the area around the London Eye is (I believe), like most of the South Bank, private property.
Chris