Your button's construction is a 1-piece brass flatbutton, with brazed-on loop, typical of early-1800s manufacture. The buttons of British army "numbered" regiments sometimes have the number in a circle surrounded by sunburst-rays. So, I'm guessing that your brass button, based on its early-1800s 1-piece construction and presumably being found on the North American continent, is a War-of-1812 era British army Regimental button. I'm not sure the number on it is 64, because the 4 (if it is actually is a 4) looks a LOT shorter than the 6.