This saying or part of a saying seems to vary enormously wherever you live. In my family if anyone asked direction my Dad would say "It's over by Bill's mother's, down by the tram sheds". The same phrase would be used for distant grey clouds. I live in the south of the country and believe it's used in the north exactly the same way. It's just a bit of family nonsense that's handed down through the generations.

The old farming types in Kent said, it meant the weather coming from the South West which is the prevailing wind direction. Who Will and his mum are I don't know, but if you consider Normandy to be in the SW, then it could refer to William the Conquerer.