I always believed it was because the decorations used to be made of natural foilage, and therefore you had natural woodland spirits in your house. This meant that they had to be let out of the house within a certain time, otherwise they were trapped and you would incur bad luck. Something like that.