Never heard about that, then again, maybe some had some visible flaws and perhaps cause harm to the shooter if fired. I know they did drop a ton if they were in battle, attacking, or retreating.
I actually came up with the answer somewhere else.
Apparently, they were not as accurate as mine balls due to the lack of the minie ball type expanding base, so experienced soldiers just tossed them aside.