Gman17 has the right of it. Most buckles that are found (especially US buckles) are due to an equipment failure, a desire to shed weight, feeling like you're more of a target with a bright shiny brass target at your waist or on your chest, even being taken prisoner and being forced to shed the belt your sidearm is strapped to. Also, another possible source is a sloppy quartermaster or sutler. The Union army was well supplied in the field. In fact, that's why when searching for a Union camp it's a good idea to pay attention to what iron you're finding. They burned crates that supplies came in and that's where many of the nails came from. I was a DIV years ago and there were several quartermaster hut holes dug that had a lot of canteens, uniform insignia, etc., still in them. Why do you suppose they just left all that behind?
If it was a grave from an unrecovered body, there would be more acroutrements (eating utensils, buttons, etc.) around it.