@Dallasb84
If you don't wash the oversize you're going to be playing catch and release with up to 10% of your gold, depending on the nature of the material.
It matters most where the rocks have a clay coating in their indentations and wrinkles. It matters less in "clean" gravels. I work on the idea that the farther from the watercourse I'm working the more need there is to wash everything thoroughly.
If you are using pans only, a big pan 17" or more can really shake and wash those oversize, it takes only a few seconds, sweep the oversize top 2-3 inches out with your hand, repeat, and then pan what has dropped to the bottom of your pan in a more conventional fashion. This is a workout.
Again with manual methods, tossing oversize into a water bucket for a final swirl around works. The bucket method is less satisfactory for getting the cleaned oversize out without dumping the nice bits at the same time! (You can do a panning action in the river at water level to roll the oversize out and keep the lowest material in the bucket) Better to dunk rocks in a perforated bucket within a slightly larger water containing bucket. Then the oversize can be discarded quite fast.
The lowest effort method I've found (again manual methods) is to put the big pan underwater in the shallows in slow-ish water, with it's riffles at the downstream side, (metal pans sink better but a plastic can be held with two rocks downstream and either side of it with a gap between them to let current through) then submerge a 1/2" classifier into it and classify the oversize into the big pan. The lights will wash away in the current, but your fines will go vertically down into the pan. A submerged classifier cleans rocks very effectively. You can work 3-4 fills of classifier and then pan what you get out once.
But all extra steps introduced into your processing reduces the amount of material washed and reduced. Let the material be the judge. Your original post suggests it may be worth your while to do something like this.