If the other driver was at fault and admitted it he should have paid your excess. The excess is for having a 'claims history'. Having made a claim even for an accident that was not your fault gives you a 'claims history' , in the view of the insurance company it means they think you are more likely to make another claim in the future, hence the excess.
Too late now to claim it back from the other driver, I would point out to Saga that they should have claimed the excess from the other driver 3 years ago and ask them to cancel it.