You need to look at your deeds. There will be sideways Ts along the boundaries.
If the T is pointing into your garden, the fence is your responsibility alone.
If it points next door, it is your neighbours. If it makes a H (so goes into both gardens) it is a party fence and you are both equally responsible.
Clearly explained here:
http://www.gardenlaw.co.uk/boundaries.html
So, if it is your fence, your neighbour should not have removed it. However, you would still be obliged to maintain it.
If it is his, he should dispose of it as he sees fit.
If it is party, you will need to come to some arrangement.
You do have a problem. If it is your neighbour's responsibility, he could erect a new fence on the boundary - this could cause your fence to be damaged or rot - you will not be able to maintain it properly.
If it is your responsibility, the neighbour is right - you have to maintain the boundary and make sure it is safe and secure.