Statistical validation of results, as Shaffer described it, simply involves testing the null hypothesis: that the pattern you detect in your data occurs at random. If you can reject the null hypothesis—and science and medicine have settled on rejecting it when there's only a five percent or less chance that it occurred at random—then you accept that your actual finding is significant.
The problem now is that we're rapidly expanding our ability to do tests. Various speakers pointed to data sources as diverse as gene expression chips and the Sloan Digital Sky Survey, which provide tens of thousands of individual data points to analyze. At the same time, the growth of computing power has meant that we can ask many questions of these large data sets at once, and each one of these tests increases the prospects than an error will occur in a study; as Shaffer put it, "every decision increases your error prospects." She pointed out that dividing data into subgroups, which can often identify susceptible subpopulations, is also a decision, and increases the chances of a spurious error. Smaller populations are also more prone to random associations.
In the end, Young noted, by the time you reach 61 tests, there's a 95 percent chance that you'll get a significant result at random. And, let's face it—researchers want to see a significant result, so there's a strong, unintentional bias towards trying different tests until something pops out.
Young went on to describe a study, published in JAMA, that was a multiple testing train wreck: exposures to 275 chemicals were considered, 32 health outcomes were tracked, and 10 demographic variables were used as controls. That was about 8,800 different tests, and as many as 9 million ways of looking at the data once the demographics were considered.
thinkahol * on 11 Jul 11Listen up, Facebook and Twitter groupies: how easily can social pressure affect your memory? Very easily, researchers at the Weizmann Institute and University College London have proved, and they think they even know what part of the brain is responsible. The participants conformed to the group on these "planted" responses, giving incorrect answers nearly 70% of the time. Volunteers watched a documentary film in small groups. Three days later, they returned to the lab individually to take a memory test, answering questions about the film. They were also asked how confident they were in their answers. They were later invited back to the lab to retake the test. This time, the subjects were also given supposed answers of the others in their film-viewing group (along with social-media-style photos) while being scanned in a functional MRI (fMRI) that revealed their brain activity. Is most of what you know false? Planted among these were false answers to questions the volunteers had previously answered correctly and confidently. The participants conformed to the group on these "planted" responses, giving incorrect answers nearly 70% of the time. To determine if their memory of the film had actually undergone a change, the researchers invited the subjects back to the lab later to take the memory test once again, telling them that the answers they had previously been fed were not those of their fellow film watchers, but random computer generations. Some of the responses reverted back to the original, correct ones, but get this: despite finding out the scientists messed with their minds, close to half of their responses remained erroneous, implying that the subjects were relying on false memories implanted in the earlier session. An analysis of the fMRI data showed a strong co-activation and connectivity between two brain areas: the hippocampus and the amygdala. Social reinforcement could act on the amygdala to persuade our brains to replace a strong memory wi
