stasys, I've looked over the link you give. Very interesting.
Ok, my comments: I work with govt. (as a subcontractor with some heavy equipment involved in asphalt/road building). So I'm a little familiar with spec's and voluminous contracts and studies like what you're saying. Uncle Sam (the military, etc...) can get downright silly with 100 page documents for something as simple erecting light-poles along the freeway, with median dividers around them. You know, how many watts the lightbulb will be, the grade steel to use in the rebar of the concrete foundation, the height of the light-pole, whether or not you used discriminatory practices in hiring the workers who erected it, and a million other points for seemingly simple tasks. And the reason for these vast blue-prints for any public works project (or military procurement of supplies, like a mine detector) is so that all the bidders are bidding on supposedly the "same thing" so that uncle same gets his mine detectors to the a) best type that he can get, and b) for the lowest price so the vendors are all vying for uncle-sam's business, and c) so that all the bidders are on the same level playing field, comparing apples to apples about what they're about to build, what spec's they'll be held to, etc... (so no one later can claim "foul play" or "un-fairness", etc..).
So I am well aware of how much volume goes into the the simplest of military procurements ($1000 toilet seats, $500 coffee pots, etc...).
To that, I would say that the end result of such careful steps, can also clarify and show the very folly I'm speaking of.
For example (don't get "lost in the example", as this is JUST an example) : I watched a documentary on the D-day landings the other night. Look at two wonderful examples of things which were tested and tested and tested before D-day, but which turned out to be total flops. AND BELIEVE ME ! charts, studies, and bid-specs were probably JUST LIKE the link you put here on T'net. It would probably would have looked TOTALLY UN-ASSAILABLE. And you and deep thought would have looked at those charts and studies and bid spec. sheets and said "see?? It can be done!". But here's the two flops: A grappling hook launched by a rocket launcher, had to be able to shoot XX feet into the air (because that was the known measured heights of the cliffs they were going to tackle). And bidders of military ware set about to design, compare, and compete for these govt. contracts. And the devices were tested and tested and tested against each other (not unlike the tests you guys want for metal detectors, for instance). And they settled on one which presumably shot the highest, provided the most durable package, and whatever else specs they needed. BUT ON D-DAY, the things failed miserably, because no one took into account that the ropes might get wet and heavy with sea-spray during the voyage over to Normandy. So you see, that option (b), that perhaps uncle same passed over in the comparisons (because it lacked a few feet in projectile distance compared to option (a), MIGHT ACTUALLY HAVE BEEN THE BETTER CHOICE. But hey, you can't argue with side-by-side testing and consumer report studies, RIGHT?
The other flop was the testing of floating tanks. Tanks that actually, believe it or not, proved themselves as do-able with many tests. Some sort of enclosure where they had a little boat-shaped cacoon, which unfolded and released the tank once it reached shore. But the seas were rough in the English channel on D-day, so most of the tanks sank
The test conditions in the months leading up to this, had been in calm bays and harbors. No one thought ahead to think "what about rough seas"?
Now I know, to answer each of the above examples of "gotchas", you might easily be thinking "well, you simply adjust the test to account for wet ropes", or "you simply re-adjust the test to account for rough seas" and so forth, right? But you have to remember that the variables, are ENDLESS. Once you close one "gotcha" loophole, another one opens.
The first mine detectors the USA used in WWII were in Africa and Italy (in the beginning of the war). There were several companies that had to compete to submit bids and prototypes, for Uncle Sam to choose from (PERHAPS NOT UNLIKE YOUR LINK SPEC'S). And the eventual model they chose was not necessarily the "most sensitive" (as you would THINK the spec's would be for). They also had to be able to be field-serviceable, inter-changeable, durable, light, green cammo paint, and a bunch of other things. And as far as "results", they only had to be able to find a frisbee sized object. And in the USA, they DID JUST THAT. Several submitted designs satisfied that, and of those, the best was chosen based on various criteria. And they worked in a) the USA test beds, b) Africa, and c) Sicily.
HOWEVER, when they got to parts of Italy (working their way northwards), they enountered places where the mine detectors weren't working too well, and no one knew why. It was eventually figured out that it was d/t ground minerals (who'd have known such a thing in those primitive days?). ALSO BY THEN the Germans had wised up to the Americans mine detectors, so they simply made subsequent land mines with less metal (plastic exterior with a lesser # of metal parts/components). And NOW the Americans mine detectors were no longer "sensitive enough". And if you take yourself back in time to the original USA test beds, you can see that it's entirely possible that another competitors model might not actually have been the SUPERIOR machine in-the-end. Perhaps anothed mine detector would find an object smaller than a frisbee. But since they didn't need to find objects that small, that became a "mute point", right? Or if one of the vendors knocked themselves silly to develop a mine detector with ground balance, perhaps he exceeded the weight limit, or the ease-of-use criteria, so they poo-pooed his detector, and chose the competitors instead.
So you can see, even in cases where you can show that such lists and charts have been done, yet history will show that it does NOTHING MORE THAN PROVE MY POINT, that there can and will still be variables that the charts can not show, or can no anticipate. But yes: Uncle Sam, in order to have a "standard" by which bidders compete, will in fact HAVE to come up with those type links and charts. Lest there can never been a criteria to attain to. Yes it serves some good (lest how else can anyone build or design or compete, if they didn't have spec's?). But for the purposes of hobbyist "what's best" or "what's deepest", no, it won't work in actual real world conditions.
