ethel is correct. it is yours to position anyay you like - tell him he is free to purchase his own fence to put up alongside yours. or even put panneling on his side, as well as hanging baskets etc, in order to make his side look nicer, but its still your fence.
sometimes people put the nice side towards their neighnours as a courtesy but there is no law or right.
i would suggest he goes halves for the fence and you let him have the nice side - saves you money and its only a fence, that you can perhaps pannel in.
also, offer him the hedge to replant on his side if he wants, to cover the bad side.