[i]why don’t they apply to France or one of the other countries they travel through[i]
Ric.ror, they do stop in France - if they've come from countries that were invaded by the French empire. You don't see great numbers of Algerians, Laotians, Djiboutians and so forth in Britain. What you see is people from countries that were taken into the British empire. That's how come they speak English rather than French.
Step one in How to Stop Immigrants Invading My Country is: "Don't invade theirs".