have you changed the mortgage into your name? if you have been able to do this he has little right to the property.
if you haven't then ask the lender or possibly a solicitor or even the C.A.B if this will be possible. if you have decent proof that you have been paying the mortage and have made significant effort in making improvements then even if he comes back you should be able at the very least to claim a good chunk of the value from sale. but look into the whole deed change issue. good luck