I often wonder about just how 'poisonous' some things actually are.
For example, most sources say that one should never eat raw potatoes, as they contain a toxin that can cause illness. However, back in my childhood days, every time my mother prepared chips for dinner, only half of them ever made it into the chip pan, as I'd invariably eaten the rest raw. I still love raw potato and it's never done me the slightest bit of harm.
The problem, of course, is in knowing which 'poisonous' things are unlikely to make many/most people ill at all and which can kill. Food writers (et al) tend to err on the side of caution and simply label them all as "poisonous". Raw potatoes probably wouldn't harm most people but raw kidney beans could possibly kill them. So, before consuming horse chestnuts, one would need to know whether the risk is at the 'potato level' or the 'kidney bean level'
Incidentally, I can't eat (genuine) sweet chestnuts at all, as the smallest piece of one is enough to give me violent stomach pains.