"Birds of a feather flock together". Immigrants head for the cheapest, poorest,areas of our cities. When Hampstead and Chelsea are full of black and brown faces we should worry that whites are being driven out because it would mean that they could afford several million for a house or flat and we couldn't. As it happens, the last three flats I owned in Chelsea and Fulham were in blocks of flats that were, or became, predominantly owned by immigrants. What's more I sold each of them to immigrants, and, by gad, they were brown, Muslim, ones as well! I don't think they moved from some flat in Hackney though, given the prices they paid :)
It is too simple to say that whites are being driven out by immigrants because these people are immigrants; the reasons are more complex and essentially economic, for betterment.
As for streets not being safe; in my lifetime, poor, white areas of our cities have had streets where it was not safe to walk unless you were keen on violence and didn't appear to have anything worth stealing.