Being a small oval instead of round, it's not a button, it is part of a cufflink. Yours appears to be made of Sterling silver. That specific "form" of cufflink (two small ovals with a loop on their back, connected by a short piece of wire) has been found in Colonial-era sites, and a few have come from civil war sites ...although most were made of plated brass, not Sterling like yours.