ask the previous neighbours if he put the fence up - as why would he put it on their property with the nice side facing them?
ask if he removed one that was already there beforehand
you are under no obligation to pay for and put up a fence.
i expect he is trying it on because he wants a new fence, so call his bluff and tell him if he wants a fence he must put one up, because you are not bothered - i'll bet he leaves his up!
he really should not have removed it until there was a replacement anyway