Read the small print.
Are your savings are instant access or is there a penalty for taking money out? Anyway you will lose the interest from the day you take the money out
Some mortgages save interest from the day you pay the money off, some are different so timing may be important. Mine changed over the time I had it.
Worth leaving some savings to give a bit of flexibility too, but of course if you do pay off the mortgage you can build up your savings with the repayments you aren't making
Is the 5% you pay gross or net.
If it is 5% before tax then you only get 4%. So either pay the mortgage off or put it in a cash isa so that you get interest tax free.
If it is in an isa then taking it out means you lose that years chance of tax free savings for ever, so I would probably leave it there.
Are you more or less confused now?
Your bank or BS probably have Financial Advisors who will not charge for advice (though they only reccomend the best their bank offer, not the absolute best). But they will help you work out the best time to pay off a lump.