To me, it looks like the broken-off disc from a "plain" Militia type tongue-&-wreath swordbelt buckle. It appears to be the right size (diameter) for that identification. Civilian tongue-&wreath buckles (shown in the photo below) were made of thin stamped-brass, and this disc is too thick to be one of those thin Civilian one, so I'm thinking it's a generic Militia plate. Several are shown in the O'Donnell & Campbell book, "American Military belt Plates." The face of the disc may have had an emblem soldered to it.
All of that being said... I do see what MIGHT be hinge-attachment remnants which DCMatt is talking about. But I don't see much at all of a thumbnail-catch for lifting a "flip-top lid (like the smoking-pipe cover has). It is reported to have come from a civil war area, and wartime Southern manufacturers were good at salvaging brass pieces'n'parts.