I think that’s true of a lot of myths, folk tales, religions, etc. Those people were eating all sorts of weird things, and there was no FDA policing the food supply. LSD can be derived from moldy rye bread, and is probably the explanation for lots of mass hallucinations over the centuries.

It seems plausible that stories of entities like fairies and elves could have arisen from experiences with substances like these mushrooms.