3 to 4 weeks only really gives you time to get to know one or two countries, rather than the 30+ countries in Europe. Europe is very large, so any attempt to fit in a large number of countries would mean that you'd probably spend more time travelling than actually visiting places. Even so, I'll attempt to answer your question.
You've not stated where you live. If you're in the UK suggesting that
London should be in your itinerary might seem either daft or obvious. However, if you're coming from the USA (or elsewhere outside Europe) then it should probably be included in your list of 'must see' places. (It's the most visited city in the world, so it must have something going for it!):
http://www.visitlondon.com/
It's unfortunate that many visitors to the UK never leave London because there's so much more of interest elsewhere. (Edinburgh, York and Bath, for example, all have loads to offer). But if you're determined to do a 'whistle stop' tour of Europe you'll probably need to look beyond the UK for your next destination.
For me,
Paris is unmissable. It's a city to fall in love with:
http://en.parisinfo.com/
While Madrid has a great deal to offer, if I had to choose one magnificent city in Spain, it would have to be
Barcelona. (Loads of history and culture but also great bars, restaurants, beaches and nightlife):
http://www.bcn.es/turisme/english/turisme/welc ome.htm
I've only visited
Rome briefly but I've liked what I've seen and it's a popular tourist destination:
http://www.romaturismo.it/v2/en/main.asp