Statistical validation of results, as Shaffer described it, simply involves testing the null hypothesis: that the pattern you detect in your data occurs at random. If you can reject the null hypothesis—and science and medicine have settled on rejecting it when there's only a five percent or less chance that it occurred at random—then you accept that your actual finding is significant.
The problem now is that we're rapidly expanding our ability to do tests. Various speakers pointed to data sources as diverse as gene expression chips and the Sloan Digital Sky Survey, which provide tens of thousands of individual data points to analyze. At the same time, the growth of computing power has meant that we can ask many questions of these large data sets at once, and each one of these tests increases the prospects than an error will occur in a study; as Shaffer put it, "every decision increases your error prospects." She pointed out that dividing data into subgroups, which can often identify susceptible subpopulations, is also a decision, and increases the chances of a spurious error. Smaller populations are also more prone to random associations.
In the end, Young noted, by the time you reach 61 tests, there's a 95 percent chance that you'll get a significant result at random. And, let's face it—researchers want to see a significant result, so there's a strong, unintentional bias towards trying different tests until something pops out.
Young went on to describe a study, published in JAMA, that was a multiple testing train wreck: exposures to 275 chemicals were considered, 32 health outcomes were tracked, and 10 demographic variables were used as controls. That was about 8,800 different tests, and as many as 9 million ways of looking at the data once the demographics were considered.
thinkahol * on 19 Aug 11I've written before about the crisis of inequality in the United States and about the quasi-religious abhorrence of "wealth redistribution" that causes many Americans to oppose tax increases, even on the ultra rich. The conviction that taxation is intrinsically evil has achieved a sadomasochistic fervor in conservative circles-producing the Tea Party, their Republican zombies, and increasingly terrifying failures of governance. Happily, not all billionaires are content to hoard their money in silence. Earlier this week, Warren Buffett published an op-ed in the New York Times in which he criticized our current approach to raising revenue. As he has lamented many times before, he is taxed at a lower rate than his secretary is. Many conservatives pretend not to find this embarrassing. Conservatives view taxation as a species of theft-and to raise taxes, on anyone for any reason, is simply to steal more. Conservatives also believe that people become rich by creating value for others. Once rich, they cannot help but create more value by investing their wealth and spawning new jobs in the process. We should not punish our best and brightest for their success, and stealing their money is a form of punishment. Of course, this is just an economic cartoon. We don't have perfectly efficient markets, and many wealthy people don't create much in the way of value for others. In fact, as our recent financial crisis has shown, it is possible for a few people to become extraordinarily rich by wrecking the global economy. Nevertheless, the basic argument often holds: Many people have amassed fortunes because they (or their parent's, parent's, parents) created value. Steve Jobs resurrected Apple Computer and has since produced one gorgeous product after another. It isn't an accident that millions of us are happy to give him our money. But even in the ideal case, where obvious value has been created, how much wealth can one person be allowed to keep? A trillion doll
