Mercury vapor is colorless and odorless. I seriously doubt those "fumes" had anything to do with the mercury from the mine. Besides the vapor being invisible cooler air from a mine escaping upward into warmer air is not a concept that normal physics can support.
We were working on the first mercury vapor detector at JIC in 1980. We field tested an undisturbed natural grassland just North of Prescott. The readings were very high. The area was not known to have significant mercury deposits before that time. There is no way to tell just where mercury vapor concentrations are high short of testing air samples at the molecular level.
Mercury does not enter the bloodstream easily through the skin, stomach or mucus membranes but mercury vapor entering the lungs is easily absorbed into the bloodstream. That's where the danger from elemental mercury lies. Touching or even eating elemental mercury has little effect but the evaporation to vapor of that same mercury occurs at room temperature and pressures and can be deadly over time in a closed space.
Keep your mercury tightly sealed and covered with water (if it's not charged) and you will be fine. Methylated mercury (organic product from some algae and bacteria) is very poisonous but it really can't exist for long in the western United States because of binding by the abundant natural element Selenium. Mercury binding with other elements in almost all cases reduces toxicity to acceptable levels.
Cinnabar is the mineral of mercury sulfide. The solubility and bioavailability of cinnabar are quite low. To get elemental mercury out of cinnabar ore it has to be roasted to temperatures well above 674 degrees Fahrenheit - the boiling point of Mercury. Because of the high temps to break out the Mercury from the sulfide form Cinnabar poisoning you in the field is not going to happen but you should be aware that elemental Mercury is sometimes found as droplets in very rich ores. You can see that in the Cinnabar deposits just south of San Francisco.
I'm not going to become a Mercury miner due to the expense of safety equipment and procedures but you are going to get more exposure from a broken light bulb that a properly equipped Mercury miner would.
Keep in mind that all minerals are potentially poisonous. Ingesting the dust or fumes from heating of minerals can be deadly. A lot more people by far have died from breathing ordinary mineral dust than from exposure to Mercury. If you are going to be grinding, crushing, heating or breathing mineral dust know what the hazards are and take the proper safety precautions. Be safe when mining or processing.
Heavy Pans