He will stare and walk away if he's busy.
I used to live in an area with a thriving Amish population and on my mom's side the family tree went that way. It's usually not too difficult to get permission, but you have to ask them at the right time and in the right way.
When I was younger I'd just ask one of the kids, if they gave you permission to hunt any of the farms it was good enough (kind of along the same lines as what Lone Star said.) Later on when I was in college I'd work some summer jobs in construction and would ask guys and always got a positive answer. Now, if I go back to visit my family in Indiana in the summer/fall, I usually stop and ask at a vegetable or pie stand (occasionally they'll have one set up at the end of their driveway.) Or occasionally, I'll go up to the house if it's midweek. If the guy doesn't come out to greet you, just leave.
The key is to not ask them when they are busy, during a social function, or on a weekend (although you can hunt then.) Socializing is basically any time a bunch of them are together (restaurants, hardware store, auctions, etc.) Don't pull into their driveway with the radio blaring, don't wear your vintage Alice Cooper t-shirt, etc.