I think it all depends on how thick the layer of gravel is. I go to a place that was an amusement park from the 1920"s-1950's. Today it's used for other things but it still has the old dirt and gravel mix parking lot. That's probably what you have under the gravel. It's really hard packed when it's dry. However, after a good rain, it really softens up where the dirt is mixed in. I would try going after a good rain which would make the dirt easier to dig in. See how deep the layer of gravel is. If there's not too much gravel on top, you may be able to push the gravel to the side with a shovel first to get down to the dirt. If there's a thick layer of gravel, it probably isn't worth it.
Glenn