Interesting. I know of two caves, basically under overhanging rocks, not like a place similar to Luray Caverns. I ain't goin in no cave! The one is documented in the history books as Revolutionary War patriots hid stuff there, the other was used to hide muskets during the Civil War. The second is only 2 miles from here. It's not big enough to hide anything in but a mountain lion.
I know of that area, I hunted Rose haven several years ago. Was hoping for a un-hunted beach but never happened. Rose Haven is a few miles south of Fair Haven. Something to think about, those beaches and cliff's? .. have eroded so much the treasures could be exposed if all was buried close to shore. I would say a 100 foot of shore erosion has happened to that area in the last 100 plus years.