This all kicked off in 1989 when an asteroid called 1989FC and another called Toutatis (a few years later) came within half a million miles of us - We only saw them after they passed!
Toutatis was 4 Km in size. If the Tunguska event
(
http://en.wikipedia.org/wiki/Tunguska_event )
was as may believe a meteorite it was about 0.5 Km
That was a 10-15 megaton explosion distroying 2,500 square Kilometers of Siberia - Imagine Toutatis hitting the US or western Europe!
That rather stopped what has been called "the giggle factor" with regards to discussion of the so called NEOs (near eartjh objects) and a lot of very serious people do loose sleep over it .
Here you can see NASA's page
http://neo.jpl.nasa.gov/risk/
you'll notice that most of the impact probabilities are e-06 or millions to one against but my aren't there a lot of them!
still most are blue (less than 50m across)
Have a look too at the close approaches page.
A Tunguska sized event probably happens every 200 years or so