Statistical validation of results, as Shaffer described it, simply involves testing the null hypothesis: that the pattern you detect in your data occurs at random. If you can reject the null hypothesis—and science and medicine have settled on rejecting it when there's only a five percent or less chance that it occurred at random—then you accept that your actual finding is significant.
The problem now is that we're rapidly expanding our ability to do tests. Various speakers pointed to data sources as diverse as gene expression chips and the Sloan Digital Sky Survey, which provide tens of thousands of individual data points to analyze. At the same time, the growth of computing power has meant that we can ask many questions of these large data sets at once, and each one of these tests increases the prospects than an error will occur in a study; as Shaffer put it, "every decision increases your error prospects." She pointed out that dividing data into subgroups, which can often identify susceptible subpopulations, is also a decision, and increases the chances of a spurious error. Smaller populations are also more prone to random associations.
In the end, Young noted, by the time you reach 61 tests, there's a 95 percent chance that you'll get a significant result at random. And, let's face it—researchers want to see a significant result, so there's a strong, unintentional bias towards trying different tests until something pops out.
Young went on to describe a study, published in JAMA, that was a multiple testing train wreck: exposures to 275 chemicals were considered, 32 health outcomes were tracked, and 10 demographic variables were used as controls. That was about 8,800 different tests, and as many as 9 million ways of looking at the data once the demographics were considered.
thinkahol * on 13 Sep 11Gilens examines the extent to which different social groups find their policy preferences reflected in actual government policy and the variation in these patterns across time and policy domains. For example, when Americans with low and high incomes disagree on policy, are policy outcomes more likely to reflect the preferences of affluent Americans? If so, does the advantage of more affluent Americans differ over time (e.g., depending on which party controls the congress and presidency) or across policy domains? Similarly, are Republicans or Democrats in the population more likely to get the policies they prefer when their party is in control of national political institutions? Because his database contains policy preferences broken down by income, education, partisanship, sex, race, region, religion, and union/non-union status, Gilens will be able to address a multitude of questions concerning government responsiveness to public preferences. For this study, Gilens uses data on public preferences and policy outcomes based on 754 national survey questions from 1992 through 1998 and restrict his attention to divergent policy preferences of low- and high-income Americans. Each of these survey questions asks respondents whether they support or oppose some proposed change in U.S. national policy, and he has used historical information sources to determine whether each proposed change occurred or not (within a four-year coding window from the date of the survey question). When Gilens looks separately at respondents with different incomes, he finds that the higher an individual's income, the more likely it is that government policy will reflect his or her preferences. This relationship, however, does not increase in a linear fashion: the difference between poor and middle-income Americans is modest compared with the difference between those with middle and high incomes. In other words, it is not that the poor are especially less likely than middle-income Americans to get
