Southie E.J., of course you are wondering how your Virginia State Seal crossbelt plate/pin was lost in Massachusetts. The theory that it was lost by a Confederate soldier in a Boston-area prisoner-of-war camp is "logical" ...but it cannot be correct. Here's why: Note that your relic was worn on a soldier's crossbelt (not a waist-belt). The crossbelt was a type of military equipment which was worn over the shoulder, across the chest. The crossbelt's purpose was to carry an ammo-pouch, or a sword-scabbard (or etc). Therefore, when a soldier got captured, all such gear was "stripped" from him. Therefore, a Confederate POW in Massachusetts would not still be wearing his crossbelt.
So, how did your relic get to Massachusetts? The most likely explanation is that it was a "battlefield souvenir" which a yankee veteran took home with him. Soldiers are notorious for taking such souvenirs from prisoners ...and an enemy's Insignia is a favorite "battle trophy" souvenir. We know such souvenir-taking occurred in the Civil War because soldiers mention doing it, in their diaries and in letters-to-home ...and because we relic-diggers have found surprising numbers of Confederate Insignia in strictly-yankee encampment sites.
We also know that long after a war, a veteran's children sometimes are given (or just "get into") daddy's (or grandpa's) old war-souvenirs ...which then "get lost in the yard." I regret to say, that's what happened to some of my dad's wartime souvenirs.
There are of course other possible explanations, but I think the info given above is the "most likely" explanation of how your civil war era Virginia State Seal crossbelt-plate/pin arrived at where you dug it.