As Bat suggests, it means nutty or eccentric. This phrase has been around for some 50 years. There is a suggestion that it may relate to enclosed ball-games - squash etc - and the erratic behaviour of a ball bouncing from a wall at a peculiar or unpredictable angle. That is, its movement takes the player by surprise, just as a "nutter's" behaviour does.