yes good advice given above. I would make a list of reasons why you would prefer him not to be alone with him, details of the anger issues you refer to and how he taken the money, and anything else relevant. Check with your citizens advice if there are any free legal services you access (some do so many hours for free per person) as they could write to him explaining your reasons and that you would be very prepare to arrange supervised access for him. Then you have evidence should he want to take it further, that you have made efforts to sort this out and have not tried to stop him seeing his child.
And i agree with salla, talks things over with a health visitor as they have experienced situations like this before and can help you access advice and look after yourself as you seem to be having a bad time of it lately
Good luck x