What you just said isn't relevant. The plan is in place, the restrictions have been set and you have to adhere to them. If this chap is keen to ensure your welfare then he will have no problem with this.
Basically, you can question it as much as you like and provide all the mitigating circumstances you want but the plan is in place, the CRB check is part of it and you've already had the answer from Social Services that you do have to do this. You might not like it but it is what it is I'm afraid.