The earliest recorded version of the phrase - meaning 'irritating me' or 'making me impatient' - in writing was in a book by C E Merriman published in 1903.
In the absence of any other evidence, that may well be where it originated.
The single word, 'nerve' had meant 'strength' or 'courage' etc for three centuries before that, so perhaps the idea was that anything that 'got on your nerves' was an irritant that sapped your strength or got you down.
Mrs Bennett in Jane Austen's Pride and Predjudice constantly refers to her nerves and that no-one takes any notice of them. This was written in the early 1800's