Good question. I would say the easiest way to differentiate them is where they are found. If it was a whale/dino stomach stone, it would be found where there are many other fossils. For example, if you found one on a river shore where there are other bones and fossils eroding from bank, I would say it was obviously a gastrolith. But if you found one in a field, midden, or shelter, I would say defiantly game ball.
If its a fossil, you'll know it. You would probably see rock formations, more fossils, and other signs. You almost never find fossils just laying on the ground, completely out of context. From what you are saying I would say you have a game ball.