First, who owns what fence may be set out in the Land Registry details. They do sometimes reveal this dangerous knowledge in the conditions which are carried over, from earlier conveyances, into the record when the property is registered; there may be enforceable terms for upkeep of the fences etc, which persist from the past and so are noted.
Second, it was conventional to mark the fences with a T mark, the T pointing towards the owner. If, by any chance, the Land Registry plan has these marks, you have your answer.
Third, the answers given by the vendor to your solicitor's enquiries before the property was conveyed to you may give the information.