I'm not sure what you mean by "It feels like it might be about to give me an electric shock". In my experience, thing give you a shock, or they don't - never heard of one that gave you a warning first!
Are these mains or battery? If mains, it's just possible that you are discharging static to earth through the casing, though as far as I know, most are double insulated and not connected to earth.
I would think that the most likely explanation is that you are feeling the vibration of the motor - in most tape machines, the motor runs continuously, even when the tape is stopped.
I have a Sony mp3 device which includes a voice recorder. it cost about 40 quid, records for hours, has no moving parts, and is slightly larger than a cigarette lighter. When I was in the UK on business recently, I recorded 52 hours of business meetings, and was able to back it up to my computer in Greece every evening over the net. Thats what I call very efficient!