As to the origin of the phrase, it first appeared in print in the 1830s in a comparison between Jane Austen and another writer. The other was said to know the 'science' of writing, whereas Austen "is like one who plays by ear". So, even in the earliest-recorded use of the phrase, it didn't specifically refer to musical ability...if you see what I mean!