Group items tagged

This week, Hannah Devlin speaks with some of the statistical vigilantes who are scouring datasets to identify cases of fraud and poor scientific practice. These include the consultant anaesthetist John Carlisle, from Torbay Hospital in Devon, who details his role in the Fujii scandal. Hannah also speaks to a PhD student from Tilburg University in the Netherlands, Michèle Nuijten, about software she has helped develop to "spell-check" statistics found in psychology papers. And finally, we hear from the University of Cambridge's Winton professor for the public understanding of risk, David Spiegelhalter, who is also president of the Royal Statistical Society, about the dangers of statistical malpractice.

Say you have a 3,000-person presidential election survey from a state where 3 percent of the population is black. If your survey is exactly representative of reality, you'd end up with 90 black people out of that 3,000. Then you ask them who they plan to vote for (for our purposes, we're assuming they're all voting). History suggests the vast majority will go with the Democrat. Over the last five presidential elections, Republicans have earned an average of only 7 percent of the black vote nationwide. However, your survey comes back with 19.5 percent of black voters leaning Republican. Now, that's the sort of unexpected result that's likely to draw the attention of a social scientist (or a curious journalist). But it should also make them suspicious. That's because when you're focusing on a tiny population like the black voters of a state with few black citizens, even a measurement error rate of 1 percent can produce an outcome that's wildly different from reality. That error could come from white voters who clicked the wrong box and misidentified their race. It could come from black voters who meant to say they were voting Democratic. In any event, the combination of an imbalanced sample ratio and measurement error can be deadly to attempts at deriving meaning from numbers - a grand piano dangling from a rope above a crenulated, four-tiered wedding cake. Just a handful of miscategorized people and - crash! - your beautiful, fascinating insight collapses into a messy disaster.