OK, no social worker. Were the police informed of his abuse to your daughter? If not why not? If you have a mental illness who do you see for help and guidance?
It could be prudent for you to contact SS yourself. Your HV or GP might be somewhere to start too. Get support to fight him
One thing I will say men who routinely threaten to 'get custody' rarely do so
Why exactly, were the other children taken into care?
As for human rights? Well, that I could say about however your child has some - to see its father - supervised visits are better than nothing for the child, not you or him, the child