Congrats on the Rifleman's button. There were a lot of southerners in the Rifleman and Dragoon troops during the 1850s. A lot of them went to the southern side when the war broke out. and even though Rifleman and Dragoons were disbanded by the time the war started, they still had their old uniforms. I have a few Rifleman and Dragoon buttons that I found at a pre-war post in West Texas. Some famous Confederates were at this post at one time or another including:
Robert E. Lee, William J Hardee, Nathan Evans, Earl Van Dorn, E. Kirby Smith, John B. Hood, and Albert Sidney Johnston.