I suspect you may have misheard and the word actually used was 'shovel-man'. That meant a labourer who used pick and shovel in the course of his work.
Alternatively, an early meaning of 'shuffle' was a trick or evasion, so maybe in some dialect a 'shuffle-man' was what we might now call a 'con-man'. (I do hope you never had such a person amongst your family!)