He won't leave you because he is a bully. He has the power over you. You may not speak to him, but do you keep the house clean? Do you cook for him? Why would he want to leave this set up? He probably has a lot of repressed guilt for years of abuse to you and the kids, and realises that nobody else will want him. You are clearly a lovely person. Go to citizens advice bureau and ask them for advice on leaving him- the house, the money etc and where you both stand. He cannot burn the house down- if he does, you can send him to jail for arson. But do tell the CAB about his threats, and preferably the police if you can, and then if he does attempt anything stupid, you will already have them on your side.
Leave him- he does not deserve to have your company. His threats are just that- threats. He sounds like a very weak and nasty man.