On a couple things, yes, you should handle with care. With most, nothing else is needed. Quartzite, granite and pyrite and fine. I've had them sitting around here for years. Asbestos, in its original matrix is ok too. The Quebec asbestos is not a problem. Now some of the African asbestos can be a problem. I have some of the Quebec stuff and it's no issue. It doesn't exactly vaporize itself and spread fumes all over the atmosphere. I have 20 pounds of it sitting around in boxes.
There are a few rules however: Do NOT lick or eat any rocks. That's rule #1. Rule #2, do not grind or saw any rock without a dust mask on, a decent one. Any low grade radioactives are safe. In general, on the East Coast you won't find many really hot rocks anymore. The mines have been reclaimed or are still producing, and the still producing part is problematic, as I'm not sure any are anymore.
Do NOT lick arsentates. Some pyrite can contain arsenic, so don't lick it. The exhaust from your car is more toxic than any rock you will probably encounter.