Agreeing... it is a Livery button, meaning, it is from a Manservant's uniform. Extremely-wealthy persons would suppy uniforms to the household's personal servants, such as butlers and coachmen, etc. Specifically, your button is a British-made one, based on its backmark seeming to say "Firmin & Sons * 153 Strand [street] * London" ...which dates your silverplated 2-piece button to approximately 1830 - 1875. (Firmin buttons from 1875 or later would imclude the abbreviation Ld or Ltd, designating its status as a Limited Liability company under British Law.)
The nine balls on your button's crown indicates the household of a Viscount. If you can tell us what the Family Motto says on the tri-fold ribbon on your button's front, we might be able to research the motto to a particular family.