Impurities. Unless you know what they are, they can be tough to remove. You'll essentially need to dissolve the impurities, but leave the gold. If this is simply carbon from the smelting, my understanding is that the old-style dry cleaning fluid - CCl4 Carbon Tetrachloride can remove it... But that stuff is nasty, and extremely hard to get a hold of.
Other folks have had some success with industrial strength toilet bowl cleaner... It won't harm the gold, but it won't be shiny when you're done.
Have you tried polishing it away? Wrap a drill bit in cloth, and use some polish on it. Then turn on the drill and slowly work the gold button around on it. That might take it off.
Best of luck to you.
Skippy