Two different schools of thought, here. Either never take off winter clothing until the month of May is ended, or until the plant 'May' is in blossom. Trouble is, May blossoms here (SE England) mid-April onwards.
I've only heard the first version - a 'clout' is an old-fashioned word for a vest, hence the 'cast', or abandon, the vest once May has been and gone. Probably both versions are true - folk sayings are, by nature, adaptble.
The may blossom one is more plausible if you ask me. May is the hawthorn bush, and is a very accurate predictor of the end of frosts. Hawthorn will not flower until all danger of frost is gone. Personally if I was still wearing vests at the end of may you could use me as a mobile sprinkler system, I would be sweating so.