In as far as how it's made, those telling you mother nature did it are correct. Both the shape and level of natural polish can be readily found in the quartzite gravels throughout central Virginia (and other areas with similar geologies). Keep in mind all the cobble/gravel quartzite here started life eons ago as quartzite outcrops on the mountain tops, which then fell down the mountains into rivers and tumbled around for a long time before being deposited and buried in the cobble and gravel beds. I have seen every kind of amazing shape imaginable and some very highly polished examples, especially from areas like here in Richmond where the pieces of stone get hung up in the granite rapids in the Fall Zone and tumble around and around.
That said, if it looks out of place, it may well have been brought there to be used as a tool. There is some pecking/wear on the small end. That could just mean that end was pointed upstream during a portion of the stone's life and got dinged up by passing gravel, or it may be use wear. The shape of the stone is very good for grinding small amounts of semi-soft material. There is that lovely natural thumb hold, and the flat end would fit nicely against the palm. Years ago when I was learning to make pine pitch glue I used a similar stone to grind up the charcoal that gets mixed in with the melted pitch. Your stone may well have been used in a similar fashion.
BTW, I did get your private message, and thank you for the kind words. If at any point you want another set of eyes to look at your site, let me know. As I said, I am in Richmond.
Keith