I always dig up everything. It never hurts to have a little bag of 'trash' so if/when you get any local authorities that aren't quite sure if you're allowed to dig things up, you can show off your bag of garbage and say, "look at all the garbage Im helping clean out of this nice park here. That can't be bad, right?"
When I lived in MD, I had a small white trashbag that I would 'pre-load' with soda cans and paper trash - on one occassion, a park ranger said, "you cant be digging holes in the park" and I showed him the trash bag and said, "this is all I've managed to dig up so far" and his response was only two words:
"carry on"