I would hazard a guess that it relates to the butchery trade, if meat was "off" it would be taken off the meat hooks in the window. It's only a guess mind.
The phrase has been around since the 1600's. The hooks referred to are the hinges of a gate, presumably the sort with rods sticking up for the gate to drop onto. "Unhinged" uses the same idea for a badly functioning mind.