Group items tagged

Statistical validation of results, as Shaffer described it, simply involves testing the null hypothesis: that the pattern you detect in your data occurs at random. If you can reject the null hypothesis—and science and medicine have settled on rejecting it when there's only a five percent or less chance that it occurred at random—then you accept that your actual finding is significant. The problem now is that we're rapidly expanding our ability to do tests. Various speakers pointed to data sources as diverse as gene expression chips and the Sloan Digital Sky Survey, which provide tens of thousands of individual data points to analyze. At the same time, the growth of computing power has meant that we can ask many questions of these large data sets at once, and each one of these tests increases the prospects than an error will occur in a study; as Shaffer put it, "every decision increases your error prospects." She pointed out that dividing data into subgroups, which can often identify susceptible subpopulations, is also a decision, and increases the chances of a spurious error. Smaller populations are also more prone to random associations. In the end, Young noted, by the time you reach 61 tests, there's a 95 percent chance that you'll get a significant result at random. And, let's face it—researchers want to see a significant result, so there's a strong, unintentional bias towards trying different tests until something pops out. Young went on to describe a study, published in JAMA, that was a multiple testing train wreck: exposures to 275 chemicals were considered, 32 health outcomes were tracked, and 10 demographic variables were used as controls. That was about 8,800 different tests, and as many as 9 million ways of looking at the data once the demographics were considered.

In his new book, "The Double Helix and the Law of Evidence" (Harvard University Press), Kaye focuses on the intersection of science and law, and emphasizes that DNA evidence is merely information. "There's a popular perception that with DNA, you get results," Kaye said. "You're either guilty or innocent, and the DNA speaks the truth. That goes too far. DNA is a tool. Perhaps in many cases it's open and shut, in other cases it's not. There's ambiguity."

One of the book's key themes is that using science in court is hard to do right. "It requires lawyers and judges to understand a lot about the science," Kaye noted. "They don't have to be scientists or technicians, but they do have to know enough to understand what's going on and whether the statements that experts are making are well-founded. The lawyers need to be able to translate that information into a form that a judge or a jury can understand." Kaye also believes that lawyers need to better understand statistics and probability, an area that has traditionally been neglected in law school curricula. His book attempts to close this gap in understanding with several sections on genetic science and probability. The book also contends that scientists, too, have contributed to the false sense of certainty, when they are so often led by either side of one particular case to take an extreme position. Scientists need to approach their role as experts less as partisans and more as defenders of truth. Aiming to be a definitive history of the use of DNA evidence, "The Double Helix and the Law of Evidence" chronicles precedent-setting criminal trials, battles among factions of the scientific community and a multitude of issues with the use of probability and statistics related to DNA. From the Simpson trial to the search for the last Russian Tsar, Kaye tells the story of how DNA science has impacted society. He delves into the history of the application of DNA science and probability within the legal system and depicts its advances and setbacks.