Statistical validation of results, as Shaffer described it, simply involves testing the null hypothesis: that the pattern you detect in your data occurs at random. If you can reject the null hypothesis—and science and medicine have settled on rejecting it when there's only a five percent or less chance that it occurred at random—then you accept that your actual finding is significant.
The problem now is that we're rapidly expanding our ability to do tests. Various speakers pointed to data sources as diverse as gene expression chips and the Sloan Digital Sky Survey, which provide tens of thousands of individual data points to analyze. At the same time, the growth of computing power has meant that we can ask many questions of these large data sets at once, and each one of these tests increases the prospects than an error will occur in a study; as Shaffer put it, "every decision increases your error prospects." She pointed out that dividing data into subgroups, which can often identify susceptible subpopulations, is also a decision, and increases the chances of a spurious error. Smaller populations are also more prone to random associations.
In the end, Young noted, by the time you reach 61 tests, there's a 95 percent chance that you'll get a significant result at random. And, let's face it—researchers want to see a significant result, so there's a strong, unintentional bias towards trying different tests until something pops out.
Young went on to describe a study, published in JAMA, that was a multiple testing train wreck: exposures to 275 chemicals were considered, 32 health outcomes were tracked, and 10 demographic variables were used as controls. That was about 8,800 different tests, and as many as 9 million ways of looking at the data once the demographics were considered.
Erich Feldmeier on 15 Nov 11"Individuals who are homozygous for the G allele of the rs53576 SNP of the oxytocin receptor (OXTR) gene tend to be more prosocial than carriers of the A allele. However, little is known about how these differences manifest behaviorally and whether they are readily detectable by outside observers, both critical questions in theoretical accounts of prosociality. In the present study, we used thin-slicing methodology to test the hypotheses that (i) individual differences in rs53576 genotype predict how prosocial observers judge target individuals to be on the basis of brief observations of behavior, and (ii) that variation in targets' nonverbal displays of affiliative cues would account for these judgment differences. In line with predictions, we found that individuals homozygous for the G allele were judged to be more prosocial than carriers of the A allele. These differences were completely accounted for by variations in the expression of affiliative cues. Thus, individual differences in rs53576 are associated with behavioral manifestations of prosociality, which ultimately guide the judgments others make about the individual. "
