In recent years I think there have been two circumstances in which this sort of experiment has been tried. The most obvious is Europe, or rather the Schengen Area, and there the results have not been disastrous as -- despite warnings to the contrary -- in the end, many people would rather stay at home than move elsewhere. Freedom of movement can bring benefits both to the individuals moving and to the countries they move to (and even from, so long as the flow of people or money isn't entirely in one direction). So it does work, although carries several problems alongside -- most notably the risk of ghettoisation and lack of proper integration between communities.
The second example is the Commonwealth. Between 1948 and 1962 pretty much everyone in the Commonwealth had the right to move the UK freely. This was a total population of something in the order of 500 - 600 million people at the time. Quite a few did come, of course, especially from the Caribbean and India, but the levels remained fairly low overall, at least compared to modern terms. (If Enoch Powell thought "we must be mad as a nation, literally mad" when only about 50,000 people per year were moving here, then I wonder what he'd say about ten times that?) Again, dire warnings turned out not to be true, as people tend to struggle to be too far away from home for long.
I expect the same would be true if the entire world opened up, and we wouldn't see Africa emptied and its inhabitants filling Europe to the brim. Several would undoubtedly come, though, and for all that I think it's right that the world should be free for all to move about as they choose, this is not something that should happen when the resulting flow of people would be so overwhelmingly in one direction. There has to be a greater level of equality first, so that people might be happy to move in either direction.