The saying is something like 'it's so cold it'd freeze the balls off a brass monkey' and has been shortened to 'it's brass monkeys' or similar to refer to the weather being very cold.
The saying has a military origin. Cannon balls used to be stacked up in pyramid like structures and these were known as brass monkeys. When it got very cold the metal would contract (my physics is a bit rusty so maybe someone could give a more scientific definition) resulting in some of the balls falling off the brass monkey.
Further to the previous answer, the Brass Monkey was actually a brass plate bolted to the desck of a wooden ship, on which the iron cannon balls were stacked.
As the damp wood expanded as the moisture froze, then the plate buckled, thus loosening the cannon balls.
The brass monkey is indeed a brass plate that used to hold stacked cannon balls. Alloys like brass tend to straighten/flatten out when cooled. Hence when it became very cold on board one of HMS's ships, the monkey's used to become flat, and could no longer hold the cannon balls. The monkeys would regain their shape when warmed up.
the answers above show that the myth concernig cannonballs has spread far & wide. the navy have never used "monkeys" they used "shot racks" when cannonballs were de rigeur. long channel contraptions.
i dont know the real answer to balls & brass monkeys - does anyone?