There is no absolute way of knowing unless you have got documentary proof eg a bill for the purchase of the fence. Note that it is also legally possible to take on responsibility for a fence, and you may still not own it, by regularly maintaining and mending it!
Here is a useful website although note that much of what they say is prefaced with “usually” or “often”
http://www.boundary-problems.co.uk/boundary-problems/fences.html
I would counsel you to avoid disagreement with your neighbour igf its at all possible as it can lead to problems when you come to sell.