I understand how this situation has grown, and how you now feel trapped and pwerless to extricate yourself from it.
Given that you wish to avoid confrontation, and that this situation is obviously causing you sonsiderable stress, i don;t think the one-off 'thanks but no thanks' approach is for you.
i would be inclined to go with grasscarp's angle, slowly but surely they will get the message if youo are simply not available. If they mail or write, ignore the contacts, if they ring, be very busy as soon as you answer, likewise if they come to the door.
This couple do appear pretty thick-skinned not to perceive your reluctance to maintain their friendship, but just because people are slow on the uptahe does not mean it is desireable to hurt their feelings - especially if that may have a residual effect, as you imply,
Just make yourself 'out of reach'. If they ring, be busy, if they write, ignore, and if they call, say it is not convenient, apologise, and close the door. They will get the message.,
Deciding that you are going to get a handle on this and take some action should calm down the stress you are feeling, which results from a sensation of helplessness.
You are not helpless, yiou have a solution, you just need tobe consistent and firm putting it into action. The results may well come sooner than you think.
Good luck.