Could be. Suggestion: Take a pair of vise grips and rip off a small piece that is sticking out. You can crush it down to powder, there are some $40 manual rock crushers on the net, I got one, and then pan the dust. A good 10x magnifier lens will tell the tale. I'm assuming it's from gold country out there. Gold colored mica will float away. Pyrite will crush to dust. Gold may deform and flatten but will not turn to dust.
I agree mica can be the culprit here. I'd certainly do some tests on it.
You can get away with a small sledge hammer and a piece of castiron plate to crush on, if you're just doing small samples for now.
Invest in a rock crusher if you need it.