The EU confers on all its members the right for their citizens to live anywhere in the EU. That's what gives me the right to live in Greece. If the UK leaves the EU, I'll no longer have that automatic right. Whether I'm then allowed to remain in Greece will be down to what the Greek government decides to do about it.
I'm not retired, but I would guess that the majority of Brits living in Europe are, so I think a mass return to the UK, might well put quite a burden on UK hospitals, social services, etc, etc.