Oh my goodness, used to do that when I was a kid! I think it was greaseproof paper folded in half around the toothed side of the comb then you blow into it.
Tissue paper's best - something like San Izal toilet roll. You fold it over the comb teeth, put it to your lips and sort of blowand hum at the same time. The sound comes from the vibration of the paper against the comb.
About fifty years ago there was a sort of instrument available in cheap shops which was made a bit like a penny whistle. It was made of plastic and looked like a miniature submarine. You blew and hummed into a spout at one end and on the top about half way along there was a mesh grill which created the sound which was very similar to the comb and paper sound.
Yes the submarine instrument Dudley refers to that sounds like a paper and comb is a "kazoo". I play kazoo myself to replicate the sax solo in Let's Stick Together with my band Men From Mars. I got mine from Early Learning Centre and it's better than the kazoos that you can get in proper music shops.