Lost in the mists of time. Wiki has a few ideas but nothing definite.
http://en.wikipedia.org/wiki/12_Days_Of_Christ mas
The Twelve Days of Christmas as a celebration and festival is old and steeped in traditions from Christian cultures around the world. For many, the idea of the 12 days as a length of celebration is based on a sequence of verses in the Holy Bible (Matthew 2:1-12), and a belief that it may have taken 12 days for the three kings (also called wise men or magi) to travel to Bethlehem. The number 12 is also symbolic of the 12 Tribes of Israel, who received Christ before the Three Kings, the Gentiles came to pay him homage
Traditionally, the twelfth day is the last day for decorations to be taken down. It is seen by many to be bad luck to take decorations down after this date, though decorations may be kept up until the next Christmas to avoid this bad luck. Those who forget to take down their decorations before the twelfth day, rarely keep them up for the remaining year though.