Statistical validation of results, as Shaffer described it, simply involves testing the null hypothesis: that the pattern you detect in your data occurs at random. If you can reject the null hypothesis—and science and medicine have settled on rejecting it when there's only a five percent or less chance that it occurred at random—then you accept that your actual finding is significant.
The problem now is that we're rapidly expanding our ability to do tests. Various speakers pointed to data sources as diverse as gene expression chips and the Sloan Digital Sky Survey, which provide tens of thousands of individual data points to analyze. At the same time, the growth of computing power has meant that we can ask many questions of these large data sets at once, and each one of these tests increases the prospects than an error will occur in a study; as Shaffer put it, "every decision increases your error prospects." She pointed out that dividing data into subgroups, which can often identify susceptible subpopulations, is also a decision, and increases the chances of a spurious error. Smaller populations are also more prone to random associations.
In the end, Young noted, by the time you reach 61 tests, there's a 95 percent chance that you'll get a significant result at random. And, let's face it—researchers want to see a significant result, so there's a strong, unintentional bias towards trying different tests until something pops out.
Young went on to describe a study, published in JAMA, that was a multiple testing train wreck: exposures to 275 chemicals were considered, 32 health outcomes were tracked, and 10 demographic variables were used as controls. That was about 8,800 different tests, and as many as 9 million ways of looking at the data once the demographics were considered.
thinkahol * on 14 Jun 11A new new brain imaging system that can identify a subject's simple thoughts may lead to clearer diagnoses for Alzheimer's disease or schizophrenia - as well as possibly paving the way for reading people's minds. Michael Greicius at Stanford University in California and colleagues used functional magnetic resonance imaging (fMRI) to identify patterns of brain activity associated with different mental states. He asked 14 volunteers to do one of four tasks: sing songs silently to themselves; recall the events of the day; count backwards in threes; or simply relax. Participants were given a 10-minute period during which they had to do this. For the rest of that time they were free to think about whatever they liked. The participants' brains were scanned for the entire 10 minutes, and the patterns of connectivity associated with each task were teased out by computer algorithms that compared scans from several volunteers doing the same task. This differs from previous experiments, in which the subjects were required to perform mental activities at specific times and the scans were then compared with brain activity when they were at rest. Greicius reasons his method encourages "natural" brain activity more like that which occurs in normal thought.
