On what basis did you work for this man?
If it was PAYE, you should have a contract of employment and payslips showing deductions for Tax and NI.
If it was casual, the businessman may have treated you as self employed, and paid you gross, cash in hand.
If it was PAYE, and you can prove it, then the employer is liable for the tax and NI which he failed to deduct or pay over, and he's probably in big trouble.
If it was casual, self employed, the Job Centre are correct.
Should be a simple matter of fact...