'score' basically means 'cut' - related to words like scar and shear. You can still talk about scoring some wood, though it sounds a bit archaic. Anyway, it came to mean a notch in something, as a way of keeping 'score', and at some stage seems to have been specifically attached to the number 20, though nobody seems quite sure why... which isn't entirely helpful, I suppose.
It probably relates to the practice in olden times of herdsmen being able to count up to 20 but little further. As a result, when counting large flocks/herds of animals, they would count that far and then - as J explains above - cut a notch in a tally-stick before starting again. Thus the farmer knew there were 84sheep if the shepherd had 4 notches and 4 sheep left over.