Group items tagged

How do statistics so often lead scientists to deny differences that those not educated in statistics can plainly see? For several generations, researchers have been warned that a statistically non-significant result does not 'prove' the null hypothesis (the hypothesis that there is no difference between groups or no effect of a treatment on some measured outcome)1. Nor do statistically significant results 'prove' some other hypothesis. Such misconceptions have famously warped the literature with overstated claims and, less famously, led to claims of conflicts between studies where none exists.

The analysis of questionnaires often involves representing the high-dimensional responses in a low-dimensional space (e.g., PCA, MCA, or t-SNE). However questionnaire data often contains categorical variables and common statistical model assumptions rarely hold. Here we present a non-parametric approach based on Fisher Information which obtains a low-dimensional embedding of a statistical manifold (SM). The SM has deep connections with parametric statistical models and the theory of phase transitions in statistical physics. Firstly we simulate questionnaire responses based on a non-linear SM and validate our method compared to other methods. Secondly we apply our method to two empirical datasets containing largely categorical variables: an anthropological survey of rice farmers in Bali and a cohort study on health inequality in Amsterdam. Compare to previous analysis and known anthropological knowledge we conclude that our method best discriminates between different behaviours, paving the way to dimension reduction as effective as for continuous data.

There are 2.5 billion people on earth without access to a toilet. That is one in three of us. When we hear the statistic, it can sound so big as to be overwhelming. Or at least impossible to solve. We'd prefer not to think about what it means. We throw up our hands. We look away.

"What these examples have in common is that they illustrate two of the key tasks of the social sciences: to measure important social variables over time and space, and to identify the social mechanisms that lead to variation in these variables. There are large problems of methodology and conceptual clarification that need to be addressed in both parts of this agenda. On the side of measurement, we have the problems of arriving at consistent and revealing definitions of economic wellbeing, using incomplete historical sources to reconstruct estimates of prices and wages, and using a range of statistical methods to validate and interpret the results. And on the explanatory side, we are faced with the difficult task of reconstructing social processes and forces in the past that may have powered the changes we are able to document, and with the task of validating the hypotheses we have put forward on the basis of historical evidence."

Physicists who think carefully about time point to troubles posed by quantum mechanics, the laws describing the probabilistic behavior of particles. At the quantum scale, irreversible changes occur that distinguish the past from the future: A particle maintains simultaneous quantum states until you measure it, at which point the particle adopts one of the states. Mysteriously, individual measurement outcomes are random and unpredictable, even as particle behavior collectively follows statistical patterns. This apparent inconsistency between the nature of time in quantum mechanics and the way it functions in relativity has created uncertainty and confusion.

I offer three specific recommendations: * Paperless DRE voting machines should be immediately phased out from US elections in favor of systems, such as precinct-counted optical scan ballots, that leave a direct artifact of the voter's choice. * Statistical "risk limiting audits" should be used after every election to detect software failures and attacks. * Additional resources, infrastructure, and training should be made available to state and local voting officials to help them more effectively defend their systems against increasingly sophisticated adversaries.

"The world is awash with bullshit, and we're drowning in it … this book is our attempt to fight back." So begins a passionate exposition of how the language of science can be weaponized to mislead both researchers and the public. Its authors are two scourges of the current 'infodemic', Carl Bergstrom and Jevin West.