I think there is confusion here. A direct debit (which the OP said it was) should result in confirmation being sent - including a "Direct Debit guarantee". It can be cancelled at any time by giving an instruction to the bank - whether the firm in whose favour it was given agrees or not (they may not even know it has been cancelled).
A continuous authority is something different. It has to be honoured by the bank (or the credit card company if it is given on a credit card) & can only be cancelled by instructing the firm holding the authority to cancel it.
It seems likely the OP did not set up a direct debit, but had unknowingly given a continuous authority. This should not happen - the firm taking it should make it clear at the time that the details will create a continuous authority. If that was not done, I suggest the OP uses RAC complaints procedure to try & get the bank charges refunded.