The reason is they gain far more from us than we do from them. If it were not thus they would be clamouring for us to leave. Our exit will see little change in the way of trade and even if it did the issue is not principally about trade or economic wellbeing but about whther people here want to live in a country called the UK or one called the EU. You will get the latter if we remain and, far from Brexit being a "leap into the unknown", it is remaining in that gives us an uncertain future. (If you doubt that, consider how the EU has changed in the last 20/30/40 years and see if you could forecast how it will change in the next 20).
The French threat to dump the Calais migrants on the UK is bluster. The Le Touquet agreement is between the UK and France. It has nothing to do with the EU. As I have explained before, it is most unlikely to be rescinded and even if it is, carriers still have the responsibility to see that all travellers have correct documentation and authority to land before they are allowed to embark. That will not change. Immigration control may be discontinued in the French ports (though it's extremely unlikely) . If it is it will be bolstered in Dover and any clandestines can be refused permission to land and returned whence they came. "Operation Fear" has obusly spread to the Continent!