JJ, most adverbs which end in 'ly' were developed from adjective forms which came first. Since I've just used the word 'first', I'll use it as an example. The earlist recorded use of first as an adjective was in 1000 AD, whereas the earliest recorded use of firstly was not until 1532. Similarly, brave 1485 and bravely 1590 and so on.
Gingerly wasn't created in this way; it did not come from the noun ginger (plant) 1000 AD or the adjective ginger (colour) 1845. Thus, your "like a ginger" analogy simply does not apply.
In fact, gingerly most probably came into English from an Old French word which gave us the now obsolete adjective, gent. This was not always an abbreviation of gentleman, but originally meant gentle or delicate.
It is easy to see, therefore, how gingerly came to mean gently, delicately or carefully.