black light, is it hard, soft? the yellow stuff looks the same only a different color. whats it look like under magnification.
Fluorite has a hardness of 4 and crystals are cubic. maybe its powdered and got wet or the building blocks that form fluorite.
I still think it looks like some mineral oxide, supergene enrichment or even microorganisms or fungus.