Toronto is one of the biggest cities in Canada so it has many common problems like any other big city, ie- pollution, noise, crime may be higher there than say a smaller city/town in comparison.
It really depends on which part of TO you settle down in, as there are safe, clean and well kept residential areas in the city as well as poorer, unsafe areas, where drugs and crime are a problem. Segregation of communities only occurs if people choose to live in those areas (ie- areas known for certain ethnicities), otherwise it's a very friendly, lively cosmopolitan city.
Toronto is probably the most expensive city (in Canada) to live in, so you could try settling in a smaller city/town ie- Kingston, Oakville, Hamilton, Ottawa etc etc, which would probably be more affordable and safer for you and your family to move in initially and get used to life in a new country.
The weather is very cold as jno mentioned (temperatures can drop to -38 degrees in Jan/Feb, worst months) ... though last couple of winters have been fairly mild in comparison (due to global warming i think)... so cold, yes but unbearable, no.
Good luck.