This phrase has been used since the 17th century to mean 'incidentally/in passing' etc. The word 'by' is being used in two different ways here...once as a preposition and once as a substantive (basically, a noun). The noun form meant a secondary subject or side issue. We still find a form of this usage in words such as 'by-road' or 'by-product'. Overall, therefore, 'by the by' just means ' by way of a subsidiary consideration'.