Go back to the CAB & ask for further advice.
From you questions you're actually dealing with an agency & the owner of the property has decided to stick his oar in - I would actually go down to your local police station - they probably won't want to know but they should be able to offer you advice on how to deal with the harassment.
You should also be able to ask for advice from your local Authority on how to deal with this agent & the house's owner.
If you're paying by Direct Debit, don't forget to inform your bank in writing (send it by registered post), you're no longer tenants with whoever the money paid to and from what they should no longer pay out the money to the agency.