Well, if it was on state park land, then technically, the answer to your question is, they probably can't metal detect there. Or I suppose if they split hairs enough, they could say they were only "casually" detecting for "modern" coins. (or were looking for meteorites, or their lost boyscout ring, etc...).
I've detected state parks and beaches before, and not had a problem. But if you're snooping around obvious historical landmarks, or making a nuisance of yourself (like big holes, as in the scene you saw), that may incite some ranger to pull rules out of a hat, that he would otherwise have never have paid attention to you.
You mention a deep hole, dust masks, and the name "smugglers canyon". I'm betting you stumbled upon some hispanic men chasing some superstition, story, etc.... Where I'm at in CA is a lot of immigrants from Mexico (d/t the agriculture/labor jobs here). So I'm well familiar with their culture, when it comes to treasures, superstition, etc... They are all convinced that you need a gas or dust mask, lest fumes from freshly excavated gold choke you to death. Kind of comical. They're all convinced there's treasures in every cave and back yard in Mexico (and I suppose any ghost story they hear here too).
