Your grandfather's button is not a Military button, nor is it from the civil war era. M.C. Lilley & Co. did not exist until 1870. It lasted from that time until around the end of World War Two... so this button can be from anytime in that range. I should mention, the Lilley Co. was not a button maker, it was a (fraternal) lodge & military goods dealer/seller. I have not seen a button with the "SERy" logo before. I think it might be what button-collectors call a Career Uniform button (hotel staff, railroad workers, etc.).
You might already know, but in case you don't:
The bullet on the left is a Confederate-made Enfield-pattern .577-caliber Minie-bullet. The one on the right is a yankee "US Regulation" .58-caliber Minie-bullet, manufactured by Williams & Co. for the US Army.