Statistical validation of results, as Shaffer described it, simply involves testing the null hypothesis: that the pattern you detect in your data occurs at random. If you can reject the null hypothesis—and science and medicine have settled on rejecting it when there's only a five percent or less chance that it occurred at random—then you accept that your actual finding is significant.
The problem now is that we're rapidly expanding our ability to do tests. Various speakers pointed to data sources as diverse as gene expression chips and the Sloan Digital Sky Survey, which provide tens of thousands of individual data points to analyze. At the same time, the growth of computing power has meant that we can ask many questions of these large data sets at once, and each one of these tests increases the prospects than an error will occur in a study; as Shaffer put it, "every decision increases your error prospects." She pointed out that dividing data into subgroups, which can often identify susceptible subpopulations, is also a decision, and increases the chances of a spurious error. Smaller populations are also more prone to random associations.
In the end, Young noted, by the time you reach 61 tests, there's a 95 percent chance that you'll get a significant result at random. And, let's face it—researchers want to see a significant result, so there's a strong, unintentional bias towards trying different tests until something pops out.
Young went on to describe a study, published in JAMA, that was a multiple testing train wreck: exposures to 275 chemicals were considered, 32 health outcomes were tracked, and 10 demographic variables were used as controls. That was about 8,800 different tests, and as many as 9 million ways of looking at the data once the demographics were considered.
David Haow on 27 Jan 19Common beans (Phaseolus vulgaris L.) are the most important legume staple food in Kenya coming second to maize. In Central Highlands of Kenya, the 0.4-0.5ton ha-1 output is below the genetic yield potential of 1.5-2ton ha-1 partly due pests and diseases. Plant parasitic nematodes (PPN) have been reported to cause yield losses of up to 60% on beans. Though bean production is important in the Central highlands of Kenya, information on PPN associated with the beans in the region is lacking. This study was therefore undertaken to establish the diversity of PPN associated with common beans and to assess the root knot nematode damage on beans in the region. The study covered 50 farms (32 in Kirinyaga and 18 in Embu Counties) distributed in eight localities namely Kibirigwi (L1), Makutano (L2), Kagio (L3), Mwea (L4) and Kutus (L5) in Kirinyaga County and Nembure (L6), Manyatta (L7) and Runyenjes (L8) in Embu County and covering three Agro Ecological Zones (AEZs); UM2 (L1, L2, L3 & L4), UM3 (L5, L7 & L8) and UM4 (L6) AEZs. Manyatta (L7) and Nembure (L6), had the highest and second highest gall indices, respectively, while Kibirigwi (L1), Makutano (L2) and Mwea (L4) had some of the lowest gall indices. The most common PPN in bean roots were Meloidogyne spp. Pratylenchus spp. and Scutellonema spp. with a frequency of 94.38%, 78.25% and 59.13%, respectively. This further confirm the importance of these nematodes in bean production systems. Upper Midland 3 (UM3) AEZs and UM4 had higher nematode population densities and diversity than UM2. Disease severity and nematode composition and distribution were notably low in the irrigated areas Kibirigwi, Kagio and Mwea compared to rain-fed areas such as Makutano, Nembure and Manyatta.
