Your 'any other car' cover is only third party.
If your policies are 'fully comp' that means that relying on the 'any other car' provision to drive your spouse's car would lose the cover on damage to that vehicle.
Further, adding another driver doesn't always increase a premium. In many cases it actually decreases it. (For example, a 17yo driver adding an experienced parent as a 'named driver' on their policy will typically see their premium CUT by 20% to 30%).
Removing the 'named driver' cover on your policies won't gain much (if anything) and could lose you a lot. Further, relying on on the 'any other car not owned by you' provision could be challenged by an insurer who might regard spouses (or partners living together as spouses) as being joint owners of their possessions.
Chris