As Dzug indicates, the fencing materials remain your neighbour's property, so he has the right to keep them.
However, you don't have to actually take them to him. You could simply advise him (preferably in writing) that he is at liberty to collect his property from you, at any reasonable time. You could further notify him that if he fails to collect his property within, say, 30 days, you will assume that he does not wish to have his property back and that you will dispose of it accordingly. (I wouldn't actually recommend burning it as soon as the 30 days are up. It would be best to write again, giving him another 7 days within which he should collect the fencing. Then double-check the legal position before striking any matches!).
You would also be entitled to charge your neighbour a reasonable storage fee (of, say, �5 per day) for looking after his fencing until he came to collect it. (However, you could only do so if you clearly stated this in your letter to him).
In practice, if your neighbour's fence has only intruded onto your land by a very short distance, it might be simplest just to put the fencing back onto his land. However, you should take care in doing so; if you damage his garden or property, you could end up having to pay him compensation.
You should also take care, when taking the fence down, to minimise any damage to it. While some damage might be inevitable, if you 'accidentally' break every fencing panel into minute pieces, your neighbour might be entitled to seek compensation from you. Equally, however, you could seek compensation (based upon a reasonable hourly rate) for the labour you have to exert in taking the fence down.
Chris