fences are not really designed to support anything, so he may not want to pay extra to have it put deeper and secured better if the problem is not his - why should he?
the extra weight of your land may actually have contributed to the fact that it fell down, so be glad he is not blaming your land.
i don't understand how it can be ''his'' fence either, surely if your land 'needs' a fence to keep it in place, the fence may be your responsibilty!
the way it faces is irrelevant - your deeds will show who is responsible for the fence.