Well as to what it is would depend on your area. There are a LOT of different materials it could be and without a picture it's hard to say.
Gold is going to "Travel" with materials that are close to it in size and weight. When you pan you're actually sorting the materials by specific gravity (S.G.) Lighter materials are going to work their way to the top while heavier ones sink to the bottom. Gold having a S.G. of a little over 19 is going to sink fast. There are other materials that have a S.G. that is almost as heavy so they are going to be right on the bottom with the gold.
Think of it this way... Water has a S.G. of ZERO Materials with a S.G. higher are going to sink in water while those with negative S.G. numbers will float on the top. Gold at 19 will weigh 19X what the same amount of water will so it sinks fast.
It's the same thing in a stream or river. The larger the gold the more water flow is required for it to move. Shape and size also come into play as well. A flake with a weight of say 2 grains is going to move easier than a small nugget of the same weight due to a larger surface area to catch more of the water.