I hunt with headphones on and avoid most questions. I'm helping clean my environment and taking home booty nobody around me sees, and that's just fine.
If they think I'm silly, or boring, or stupid, it is obvious to me they aren't thinking very hard. I'm not rooting through garbage cans (which can also be profitable--especially in Southern California) and I'm not causing trouble. I'm bored with TV and don't go in much for social events. Actually, one of the things I like about hunting is I don't have to answer to anyone. If they heckle me I just move on in my own direction at my own pace.
One time I was finding pennies and pull tabs, and some teens started harrassing me, asking if I had found gold, and digging with their hands and saying "nope, no gold here", over and over. I ignored them completely and when they turned their backs I dug up a huge crucifix on a nice silver chain. Boy, am I glad they didn't see it!
No matter where you go, people generally have the same types of thinking patterns, and rejecting detectorists is somewhat ubiquitous. Those people I ignore, but not everyone feels that way.
Another time I saw a kid looking at my detector, and from a hundred yards away he just shouted "I WANT one of those!" Smart kid!