//Even shying away from rejoining the single market.//
The EU has a few institutions as its basis. The most important are the Single Currency, freedom of movement, the Single Market and the Customs Union. Compliance with all of those is mandatory for membership. So, as a basis for EU membership, no nation could say "We'll be a member, but we'll opt out of the Single Market".
When the Brexit vote was secured, because it was a small majority in favour of leaving and during the interminable negotiations it was often suggested that some concessions should be made to appease the disappointed 48%. Among them was leaving the EU but remaining in the Single Market. I argued against that on the basis that, had the result been the other way round, it would have been ludicrous to suggest that a concession (such as remaining as a member of the EU but leaving the Single Market) could be made for the disappointed Leavers. I stand by that. The EU and the Single Market are synonymous. I know there are four countries which are members of the Single Market but are not EU members: Iceland, Liechtenstein, and Norway (as members of the European Economic Area) and Switzerland through sectoral treaties. The agreements which bind the last of these to the SM are under threat - the EU does not like sectoral agreements, it prefers all-encompassing treaties (and all enveloping control). The continued SM membership of Switzerland is by no means guaranteed. Iceland has a population smaller than the London Borough of Croydon and Liechtenstein one that would leave a Premiership football club wondering where their fans had gone on match day. Norway (or at least its politicians) always had ambitions to join the EU, but the electorate twice rejected the idea. But by then they had become embroiled with the EU by way of the EFTA and the die was cast for them to be subject to around 20% of EU laws in order for them to freely sell electricity to EU members.
In short, there is no justification for a large nation that is not an EU member to join the Single Market. It would be subject to most of the more significant EU legislation without having even the limited influence that member nations have over the domineering EU Commission.