Dealings with HMRC are heavily stacking in their favour, sadly. In an overpayment case the default position, that is probably still very difficult to challenge, is that it was almost certainly not their fault (this despite having had tens of thousands of such problems, itself probably an underestimate) so you will have to pay back the excess (and sometimes even more, although the rules may have changed).
Our own family had such a case once. My mother is not exactly a novice when it comes to government forms and understanding benefits claims, but nevertheless we had a massive overpayment about ten years back. Eventually, this was tracked down to some administrative error whereby HMRC decided that it didn't know what her annual salary was in the new year, but decided to put down what was described, without irony, as a "reasonable estimate". Their chosen estimate for her total annual salary before tax ... £1.
Mum probably knew the system almost as well as, if not better than, the people on the other side of the phone, and it still didn't make much difference. We ended up having to pay back the overpayment and more besides.
I know this doesn't bode well for the OP's chances, but on the other hand this was ten years ago or so, and things may have changed. The advice given of getting in touch with the CAB I'd endorse -- they will have the skills needed to deal with this best.