I'll try to avoid the squabbling and simply point out the difference between a 'direct debit' and a 'continuous authority'.
If you set up a direct debit to pay someone you are protected by the 'Direct Debit Guarantee'. The recipient of the money has to tell you in advance how much they'll be taking from your account. If they fail to do so, your bank is obliged to repay the money to you. You can also cancel the direct debit at any time. If your bank then pays out any money, under the direct debit, they're obliged to refund it to you.
However giving someone your debit card details is completely different. You're providing a 'continuous authority' which allows the recipient to take any money which is lawfully theirs. You can't cancel the arrangement. (Only the recipient can do that). Further, your bank is legally obliged to pay out any money demanded under that continuous authority.
So, regrettably, it's unlikely that there's anything you can do about the situation. You bank had no choice other than to hand over the money (even if you'd contacted therm beforehand and asked them not to do so) and the people you owed the money to had the legal right to take it.
Chris