My apologies for a very wordy answer. There have been a decent number of studies on serrations vs strait edges, and I think it varies between knives and projectile points, and then it varies with the animals hunted. Here are my thoughts:
For true projectile points. There are number of potential advantages to serrations but on deer and elk sized targets the combination of a decent bow, a good arrow shaft and an experienced hunter will outweigh any differences between a bone arrowhead, a straight edge stone point, a serrated stone point, a larger or smaller point, or even a fire hardened pointed arrow shaft. The modern equivalent would be an experienced hunter with a decently powerful rifle they know and know how to shoot. A friend of mine here in Florida has probably taken a couple hundred hogs with a bolt action 30-06 on his farm over the years. It isn’t going to make a bit of difference if he shoots a 50 cent bullet or a $5 bullet on a hog inside of 75 yards.
A serrated point has more cutting edge which is good. A serrated point can be wider than straightedge point of the same weight, and that equals a bigger hole which can be good. (It also means less penetration, so depending on the animal that might not be good.) I’ve also seen some interesting impact fractures on serrated points that might suggest that bigger serrations help an arrow glance off bone better while continuing on a slightly altered trajectory. (The serration breaks along the side, but the point doesn’t shatter.)
I’ve had a lot of chats about the serrations cutting up the animal as the arrow moves as the animal runs. I think again it probably depends on the animal. Deer tend to bed down within 200 yards or so of being shot if they aren’t being chased, so for them the shot is usually a kill shot or not after the initial impact. Elk will run more, but I think natives used dogs a lot more than we would during hunting so they could probably track the animal faster. (Give it a shot to die before they tracked it, but they had to risk pushing the animal further than we do today. If they didn’t recover it within hours, wolves and other animals would get it.) In either case, counting on the arrowhead to cut up internal organs or arteries probably doesn’t make much of a difference on shorter hunts.
It might have made a huge difference to plains hunters taking multiple atlatl shots on multiple bison from a distance. I think they were simply trying to hit a couple of animals broad side (gut shots welcome) and had days’ worth of open plains to track them. Maybe it was also a big deal to Clovis hunters taking down hairy elephants with multiple shots.
All of that said, I think most serrations were aesthetic, they look good.
For true knives, I think there is a component of the cutting edge, durability, and loss of material when resharpening. The high points on a serrated knife can get dull, and the knife still has cutting power because the cutting edge on the inside can still be sharp and protected. Most groups still butchered big animals and bulk kills with strait edge bladelets because you get so much more cutting edge out of a cobble of material, but a serrated biface was probably more versatile on a solo hunt. It would also work for other stuff like scraping bark off wood for kindling and other tools uses like that (what Monsterrack mentioned), and it would get through bones and joints better. (A little more swiss army knife like.) That said, the big benefit was probably that you lose less material when resharpening the knife by serrating it. If you want to quickly resharpen a knife, you can just take pressure flakes skipping spaces and you get a sharp edge quickly without reducing the width of the knife much. A bevel is a better edge, but it reduces the width more. A bifacial resharpening job reduces the width even more. Again, like points there seems to come a point where some groups liked the look of serrations because they put them on patterns and even over worked them so they would be more prominent.
For early archaic points that were probably darts and knives, serrations were more versatile and they were hitting smaller game with a very powerful projectile, so the knife like benefits didn't hurt the projectile part of the equation.
