Skip to main content

Home/ New Media Ethics 2009 course/ Group items tagged Methodology

Rss Feed Group items tagged

Weiye Loh

Effect of alcohol on risk of coronary heart diseas... [Vasc Health Risk Manag. 2006] - ... - 0 views

  • Studies of the effects of alcohol consumption on health outcomes should recognise the methodological biases they are likely to face, and design, analyse and interpret their studies accordingly. While regular moderate alcohol consumption during middle-age probably does reduce vascular risk, care should be taken when making general recommendations about safe levels of alcohol intake. In particular, it is likely that any promotion of alcohol for health reasons would do substantially more harm than good.
  • . The consistency in the vascular benefit associated with moderate drinking (compared with non-drinking) observed across different studies, together with the existence of credible biological pathways, strongly suggests that at least some of this benefit is real.
  • However, because of biases introduced by: choice of reference categories; reverse causality bias; variations in alcohol intake over time; and confounding, some of it is likely to be an artefact. For heavy drinking, different study biases have the potential to act in opposing directions, and as such, the true effects of heavy drinking on vascular risk are uncertain. However, because of the known harmful effects of heavy drinking on non-vascular mortality, the problem is an academic one.
  •  
    Studies of the effects of alcohol consumption on health outcomes should recognise the methodological biases they are likely to face, and design, analyse and interpret their studies accordingly. While regular moderate alcohol consumption during middle-age probably does reduce vascular risk, care should be taken when making general recommendations about safe levels of alcohol intake.
Weiye Loh

Sam Harris: Toward a Science of Morality - 0 views

  • What about depression? Is it impossible to define or study this state of mind empirically? I'm not sure how deep Carroll's skepticism runs, but much of psychology now appears to hang in the balance. Of course, Carroll might want to say that the problem of access to the data of first-person experience is what makes psychology often seem to teeter at the margin of science. He might have a point -- but, if so, it would be a methodological point, not a point about the limits of scientific truth. Remember, the science of determining exactly which books were in the Library of Alexandria is stillborn and going absolutely nowhere, methodologically speaking. But this doesn't mean we can't be absolutely right or absolutely wrong about the relevant facts.
    • Weiye Loh
       
      What kind of science are we discussing if there's no methodology? Popperian? Certainly not Kuhnian. 
  • While I'm happy to admit that people are morally confused, I see no evidence whatsoever that they all ultimately want the same thing. The position doesn't even seem coherent. Is it a priori necessary that people ultimately have the same idea about human well-being, or is it a contingent truth about actual human beings?
  • I might find that brain state X242358B is my absolute favorite, and Carroll might prefer X979793L, but the fear that we will radically diverge in our judgments about what constitutes well-being seems pretty far-fetched. The possibility that my hell will be someone else's heaven, and vice versa, seems hardly worth considering. And yet, whatever divergence did occur must also depend on facts about the brains in question.
  •  
    Toward a Science of Morality Sam HarrisPosted: May 7, 2010 12:47 AM
Weiye Loh

Rationally Speaking: Should non-experts shut up? The skeptic's catch-22 - 0 views

  • You can read the talk here, but in a nutshell, Massimo was admonishing skeptics who reject the scientific consensus in fields in which they have no technical expertise - the most notable recent example of this being anthropogenic climate change, about which venerable skeptics like James Randi and Michael Shermer have publicly expressed doubts (though Shermer has since changed his mind).
  • I'm totally with Massimo that it seems quite likely that anthropogenic climate change is really happening. But I'm not sure I can get behind Massimo's broader argument that non-experts should defer to the expert consensus in a field.
  • First of all, while there are strong incentives for a researcher to find errors in other work in the field, there are strong disincentives for her to challenge the field's foundational assumptions. It will be extremely difficult for her to get other people to agree with her if she tries, and if she succeeds, she'll still be taking herself down along with the rest of the field.
  • ...7 more annotations...
  • Second of all, fields naturally select for people who accept their foundational assumptions. People who don't accept those assumptions are likely not to have gone into that field in the first place, or to have left it already.
  • Sometimes those foundational assumptions are simple enough that an outsider can evaluate them - for instance, I may not be an expert in astrology or theology, but I can understand their starting premises (stars affect human fates; we should accept the Bible as the truth) well enough to confidently dismiss them, and the fields that rest on them. But when the foundational assumptions get more complex - like the assumption that we can reliably model future temperatures - it becomes much harder for an outsider to judge their soundness.
  • we almost seem to be stuck in a Catch-22: The only people who are qualified to evaluate the validity of a complex field are the ones who have studied that field in depth - in other words, experts. Yet the experts are also the people who have the strongest incentives not to reject the foundational assumptions of the field, and the ones who have self-selected for believing those assumptions. So the closer you are to a field, the more biased you are, which makes you a poor judge of it; the farther away you are, the less relevant knowledge you have, which makes you a poor judge of it. What to do?
  • luckily, the Catch-22 isn't quite as stark as I made it sound. For example, you can often find people who are experts in the particular methodology used by a field without actually being a member of the field, so they can be much more unbiased judges of whether that field is applying the methodology soundly. So for example, a foundational principle underlying a lot of empirical social science research is that linear regression is a valid tool for modeling most phenomena. I strongly recommend asking a statistics professor about that. 
  • there are some general criteria that outsiders can use to evaluate the validity of a technical field, even without “technical scientific expertise” in that field. For example, can the field make testable predictions, and does it have a good track record of predicting things correctly? This seems like a good criterion by which an outsider can judge the field of climate modeling (and "predictions" here includes using your model to predict past data accurately). I don't need to know how the insanely-complicated models work to know that successful prediction is a good sign.
  • And there are other more field-specific criteria outsiders can often use. For example, I've barely studied postmodernism at all, but I don't have to know much about the field to recognize that the fact that they borrow concepts from complex disciplines which they themselves haven't studied is a red flag.
  • the issue with AGW is less the science and all about the political solutions. Most every solution we hear in the public conversation requires some level of sacrifice and uncertainty in the future.Politicians, neither experts in climatology nor economics, craft legislation to solve the problem through the lens of their own political ideology. At TAM8, this was pretty apparent. My honest opinion is that people who are AGW skeptics are mainly skeptics of the political solutions. If AGW was said to increase the GDP of the country by two to three times, I'm guessing you'd see a lot less climate change skeptics.
  •  
    WEDNESDAY, JULY 14, 2010 Should non-experts shut up? The skeptic's catch-22
Weiye Loh

Science Warriors' Ego Trips - The Chronicle Review - The Chronicle of Higher Education - 0 views

  • By Carlin Romano Standing up for science excites some intellectuals the way beautiful actresses arouse Warren Beatty, or career liberals boil the blood of Glenn Beck and Rush Limbaugh. It's visceral.
  • A brave champion of beleaguered science in the modern age of pseudoscience, this Ayn Rand protagonist sarcastically derides the benighted irrationalists and glows with a self-anointed superiority. Who wouldn't want to feel that sense of power and rightness?
  • You hear the voice regularly—along with far more sensible stuff—in the latest of a now common genre of science patriotism, Nonsense on Stilts: How to Tell Science From Bunk (University of Chicago Press), by Massimo Pigliucci, a philosophy professor at the City University of New York.
  • ...24 more annotations...
  • it mixes eminent common sense and frequent good reporting with a cocksure hubris utterly inappropriate to the practice it apotheosizes.
  • According to Pigliucci, both Freudian psychoanalysis and Marxist theory of history "are too broad, too flexible with regard to observations, to actually tell us anything interesting." (That's right—not one "interesting" thing.) The idea of intelligent design in biology "has made no progress since its last serious articulation by natural theologian William Paley in 1802," and the empirical evidence for evolution is like that for "an open-and-shut murder case."
  • Pigliucci offers more hero sandwiches spiced with derision and certainty. Media coverage of science is "characterized by allegedly serious journalists who behave like comedians." Commenting on the highly publicized Dover, Pa., court case in which U.S. District Judge John E. Jones III ruled that intelligent-design theory is not science, Pigliucci labels the need for that judgment a "bizarre" consequence of the local school board's "inane" resolution. Noting the complaint of intelligent-design advocate William Buckingham that an approved science textbook didn't give creationism a fair shake, Pigliucci writes, "This is like complaining that a textbook in astronomy is too focused on the Copernican theory of the structure of the solar system and unfairly neglects the possibility that the Flying Spaghetti Monster is really pulling each planet's strings, unseen by the deluded scientists."
  • Or is it possible that the alternate view unfairly neglected could be more like that of Harvard scientist Owen Gingerich, who contends in God's Universe (Harvard University Press, 2006) that it is partly statistical arguments—the extraordinary unlikelihood eons ago of the physical conditions necessary for self-conscious life—that support his belief in a universe "congenially designed for the existence of intelligent, self-reflective life"?
  • Even if we agree that capital "I" and "D" intelligent-design of the scriptural sort—what Gingerich himself calls "primitive scriptural literalism"—is not scientifically credible, does that make Gingerich's assertion, "I believe in intelligent design, lowercase i and lowercase d," equivalent to Flying-Spaghetti-Monsterism? Tone matters. And sarcasm is not science.
  • The problem with polemicists like Pigliucci is that a chasm has opened up between two groups that might loosely be distinguished as "philosophers of science" and "science warriors."
  • Philosophers of science, often operating under the aegis of Thomas Kuhn, recognize that science is a diverse, social enterprise that has changed over time, developed different methodologies in different subsciences, and often advanced by taking putative pseudoscience seriously, as in debunking cold fusion
  • The science warriors, by contrast, often write as if our science of the moment is isomorphic with knowledge of an objective world-in-itself—Kant be damned!—and any form of inquiry that doesn't fit the writer's criteria of proper science must be banished as "bunk." Pigliucci, typically, hasn't much sympathy for radical philosophies of science. He calls the work of Paul Feyerabend "lunacy," deems Bruno Latour "a fool," and observes that "the great pronouncements of feminist science have fallen as flat as the similarly empty utterances of supporters of intelligent design."
  • It doesn't have to be this way. The noble enterprise of submitting nonscientific knowledge claims to critical scrutiny—an activity continuous with both philosophy and science—took off in an admirable way in the late 20th century when Paul Kurtz, of the University at Buffalo, established the Committee for the Scientific Investigation of Claims of the Paranormal (Csicop) in May 1976. Csicop soon after launched the marvelous journal Skeptical Inquirer
  • Although Pigliucci himself publishes in Skeptical Inquirer, his contributions there exhibit his signature smugness. For an antidote to Pigliucci's overweening scientism 'tude, it's refreshing to consult Kurtz's curtain-raising essay, "Science and the Public," in Science Under Siege (Prometheus Books, 2009, edited by Frazier)
  • Kurtz's commandment might be stated, "Don't mock or ridicule—investigate and explain." He writes: "We attempted to make it clear that we were interested in fair and impartial inquiry, that we were not dogmatic or closed-minded, and that skepticism did not imply a priori rejection of any reasonable claim. Indeed, I insisted that our skepticism was not totalistic or nihilistic about paranormal claims."
  • Kurtz combines the ethos of both critical investigator and philosopher of science. Describing modern science as a practice in which "hypotheses and theories are based upon rigorous methods of empirical investigation, experimental confirmation, and replication," he notes: "One must be prepared to overthrow an entire theoretical framework—and this has happened often in the history of science ... skeptical doubt is an integral part of the method of science, and scientists should be prepared to question received scientific doctrines and reject them in the light of new evidence."
  • Pigliucci, alas, allows his animus against the nonscientific to pull him away from sensitive distinctions among various sciences to sloppy arguments one didn't see in such earlier works of science patriotism as Carl Sagan's The Demon-Haunted World: Science as a Candle in the Dark (Random House, 1995). Indeed, he probably sets a world record for misuse of the word "fallacy."
  • To his credit, Pigliucci at times acknowledges the nondogmatic spine of science. He concedes that "science is characterized by a fuzzy borderline with other types of inquiry that may or may not one day become sciences." Science, he admits, "actually refers to a rather heterogeneous family of activities, not to a single and universal method." He rightly warns that some pseudoscience—for example, denial of HIV-AIDS causation—is dangerous and terrible.
  • But at other points, Pigliucci ferociously attacks opponents like the most unreflective science fanatic
  • He dismisses Feyerabend's view that "science is a religion" as simply "preposterous," even though he elsewhere admits that "methodological naturalism"—the commitment of all scientists to reject "supernatural" explanations—is itself not an empirically verifiable principle or fact, but rather an almost Kantian precondition of scientific knowledge. An article of faith, some cold-eyed Feyerabend fans might say.
  • He writes, "ID is not a scientific theory at all because there is no empirical observation that can possibly contradict it. Anything we observe in nature could, in principle, be attributed to an unspecified intelligent designer who works in mysterious ways." But earlier in the book, he correctly argues against Karl Popper that susceptibility to falsification cannot be the sole criterion of science, because science also confirms. It is, in principle, possible that an empirical observation could confirm intelligent design—i.e., that magic moment when the ultimate UFO lands with representatives of the intergalactic society that planted early life here, and we accept their evidence that they did it.
  • "As long as we do not venture to make hypotheses about who the designer is and why and how she operates," he writes, "there are no empirical constraints on the 'theory' at all. Anything goes, and therefore nothing holds, because a theory that 'explains' everything really explains nothing."
  • Here, Pigliucci again mixes up what's likely or provable with what's logically possible or rational. The creation stories of traditional religions and scriptures do, in effect, offer hypotheses, or claims, about who the designer is—e.g., see the Bible.
  • Far from explaining nothing because it explains everything, such an explanation explains a lot by explaining everything. It just doesn't explain it convincingly to a scientist with other evidentiary standards.
  • A sensible person can side with scientists on what's true, but not with Pigliucci on what's rational and possible. Pigliucci occasionally recognizes that. Late in his book, he concedes that "nonscientific claims may be true and still not qualify as science." But if that's so, and we care about truth, why exalt science to the degree he does? If there's really a heaven, and science can't (yet?) detect it, so much the worse for science.
  • Pigliucci quotes a line from Aristotle: "It is the mark of an educated mind to be able to entertain a thought without accepting it." Science warriors such as Pigliucci, or Michael Ruse in his recent clash with other philosophers in these pages, should reflect on a related modern sense of "entertain." One does not entertain a guest by mocking, deriding, and abusing the guest. Similarly, one does not entertain a thought or approach to knowledge by ridiculing it.
  • Long live Skeptical Inquirer! But can we deep-six the egomania and unearned arrogance of the science patriots? As Descartes, that immortal hero of scientists and skeptics everywhere, pointed out, true skepticism, like true charity, begins at home.
  • Carlin Romano, critic at large for The Chronicle Review, teaches philosophy and media theory at the University of Pennsylvania.
  •  
    April 25, 2010 Science Warriors' Ego Trips
Weiye Loh

Adventures in Flay-land: Dealing with Denialists - Delingpole Part III - 0 views

  • This post is about how one should deal with a denialist of Delingpole's ilk.
  • I saw someone I follow on Twitter retweet an update from another Twitter user called @AGW_IS_A_HOAX, which was this: "NZ #Climate Scientists Admit Faking Temperatures http://bit.ly/fHbdPI RT @admrich #AGW #Climategate #Cop16 #ClimateChange #GlobalWarming".
  • So I click on it. And this is how you deal with a denialist claim. You actually look into it. Here is the text of that article reproduced in full: New Zealand Climate Scientists Admit To Faking Temperatures: The Actual Temps Show Little Warming Over Last 50 YearsRead here and here. Climate "scientists" across the world have been blatantly fabricating temperatures in hopes of convincing the public and politicians that modern global warming is unprecedented and accelerating. The scientists doing the fabrication are usually employed by the government agencies or universities, which thrive and exist on taxpayer research dollars dedicated to global warming research. A classic example of this is the New Zealand climate agency, which is now admitting their scientists produced bogus "warming" temperatures for New Zealand. "NIWA makes the huge admission that New Zealand has experienced hardly any warming during the last half-century. For all their talk about warming, for all their rushed invention of the “Eleven-Station Series” to prove warming, this new series shows that no warming has occurred here since about 1960. Almost all the warming took place from 1940-60, when the IPCC says that the effect of CO2 concentrations was trivial. Indeed, global temperatures were falling during that period.....Almost all of the 34 adjustments made by Dr Jim Salinger to the 7SS have been abandoned, along with his version of the comparative station methodology."A collection of temperature-fabrication charts.
  • ...10 more annotations...
  • I check out the first link, the first "here" where the article says "Read here and here". I can see that there's been some sort of dispute between two New Zealand groups associated with climate change. One is New Zealand’s Climate Science Coalition (NZCSC) and the other is New Zealand’s National Institute of Water and Atmospheric Research (NIWA), but it doesn't tell me a whole lot more than I already got from the other article.
  • I check the second source behind that article. The second article, I now realize, is published on the website of a person called Andrew Montford with whom I've been speaking recently and who is the author of a book titled The Hockey Stick Illusion. I would not label Andrew a denialist. He makes some good points and seems to be a decent guy and geniune sceptic (This is not to suggest all denialists are outwardly dishonest; however, they do tend to be hard to reason with). Again, this article doesn't give me anything that I haven't already seen, except a link to another background source. I go there.
  • From this piece written up on Scoop NZNEWSUK I discover that a coalition group consisting of the NZCSC and the Climate Conversation Group (CCG) has pressured the NIWA into abandoning a set of temperature record adjustments of which the coalition dispute the validity. This was the culmination of a court proceeding in December 2010, last month. In dispute were 34 adjustments that had been made by Dr Jim Salinger to the 7SS temperature series, though I don't know what that is exactly. I also discover that there is a guy called Richard Treadgold, Convenor of the CCG, who is quoted several times. Some of the statements he makes are quoted in the articles I've already seen. They are of a somewhat snide tenor. The CSC object to the methodology used by the NIWA to adjust temperature measurements (one developed as part of a PhD thesis), which they critique in a paper in November 2009 with the title "Are we feeling warmer yet?", and are concerned about how this public agency is spending its money. I'm going to have to dig a bit deeper if I want to find out more. There is a section with links under the heading "Related Stories on Scoop". I click on a few of those.
  • One of these leads me to more. Of particular interest is a fairly neutral article outlining the progress of the court action. I get some more background: For the last ten years, visitors to NIWA’s official website have been greeted by a graph of the “seven-station series” (7SS), under the bold heading “New Zealand Temperature Record”. The graph covers the period from 1853 to the present, and is adorned by a prominent trend-line sloping sharply upwards. Accompanying text informs the world that “New Zealand has experienced a warming trend of approximately 0.9°C over the past 100 years.” The 7SS has been updated and used in every monthly issue of NIWA’s “Climate Digest” since January 1993. Its 0.9°C (sometimes 1.0°C) of warming has appeared in the Australia/NZ Chapter of the IPCC’s 2001 and 2007 Assessment Reports. It has been offered as sworn evidence in countless tribunals and judicial enquiries, and provides the historical base for all of NIWA’s reports to both Central and Local Governments on climate science issues and future projections.
  • now I can see why this is so important. The temperature record informs the conclusions of the IPCC assessment reports and provides crucial evidence for global warming.
  • Further down we get: NIWA announces that it has now completed a full internal examination of the Salinger adjustments in the 7SS, and has forwarded its “review papers” to its Australian counterpart, the Bureau of Meteorology (BOM) for peer review.and: So the old 7SS has already been repudiated. A replacement NZTR [New Zealand Temperature Record] is being prepared by NIWA – presumably the best effort they are capable of producing. NZCSC is about to receive what it asked for. On the face of it, there’s nothing much left for the Court to adjudicate.
  • NIWA has been forced to withdraw its earlier temperature record and replace it with a new one. Treadgold quite clearly states that "NIWA makes the huge admission that New Zealand has experienced hardly any warming during the last half-century" and that "the new temperature record shows no evidence of a connection with global warming." Earlier in the article he also stresses the role of the CSC in achieving these revisions, saying "after 12 months of futile attempts to persuade the public, misleading answers to questions in the Parliament from ACT and reluctant but gradual capitulation from NIWA, their relentless defence of the old temperature series has simply evaporated. They’ve finally given in, but without our efforts the faulty graph would still be there."
  • All this leads me to believe that if I look at the website of NIWA I will see a retraction of the earlier position and a new position that New Zealand has experienced no unusual warming. This is easy enough to check. I go there. Actually, I search for it to find the exact page. Here is the 7SS page on the NIWA site. Am I surprised that NIWA have retracted nothing and that in fact their revised graph shows similar results? Not really. However, I am somewhat surprised by this page on the Climate Conversation Group website which claims that the 7SS temperature record is as dead as the parrot in the Monty Python sketch. It says "On the eve of Christmas, when nobody was looking, NIWA declared that New Zealand had a new official temperature record (the NZT7) and whipped the 7SS off its website." However, I've already seen that this is not true. Perhaps there was once a 7SS graph and information about the temperature record on the site's homepage that can no longer be seen. I don't know. I can only speculate. I know that there is a section on the NIWA site about the 7SS temperature record that contains a number of graphs and figures and discusses recent revisions. It has been updated as recently as December 2010, last month. The NIWA page talks all about the 7SS series and has a heading that reads "Our new analysis confirms the warming trend".
  • The CCG page claims that the new NZT7 is not in fact a revision but rather a replacement. Although it results in a similar curve, the adjustments that were made are very different. Frankly I can't see how that matters at the end of the day. Now, I don't really know whether I can believe that the NIWA analysis is true, but what I am in no doubt of whatsoever is that the statements made by Richard Treadgold that were quoted in so many places are at best misleading. The NIWA has not changed its position in the slightest. The assertion that the NIWA have admitted that New Zealand has not warmed much since 1960 is a politician's careful argument. Both analyses showed the same result. This is a fact that NIWA have not disputed; however, they still maintain a connection to global warming. A document explaining the revisions talks about why the warming has slowed after 1960: The unusually steep warming in the 1940-1960 period is paralleled by an unusually large increase in northerly flow* during this same period. On a longer timeframe, there has been a trend towards less northerly flow (more southerly) since about 1960. However, New Zealand temperatures have continued to increase over this time, albeit at a reduced rate compared with earlier in the 20th century. This is consistent with a warming of the whole region of the southwest Pacific within which New Zealand is situated.
  • Denialists have taken Treadgold's misleading mantra and spread it far and wide including on Twitter and fringe websites, but it is faulty as I've just demonstrated. Why do people do this? Perhaps they are hoping that others won't check the sources. Most people don't. I hope this serves as a lesson for why you always should.
Weiye Loh

Rationally Speaking: Between scientists and citizens, part I - 0 views

  • The authors suggest that there are two publics for science communication, one that is liberal, educated and with a number of resources at its disposals; the other with less predictable and less-formed opinions. The authors explored empirically (via a survey of 108 Colorado citizens) the responses of liberal and educated people to scientific jargon by exposing them to two “treatments”: jargon-laden vs lay terminology news articles. The results found that scientists were considered the most credible sources in the specific area of environmental science (94.3% agreed), followed by activists (61.1%). The least credible were industry representatives, clergy and celebrities. (Remember, this is among liberal educated people.) Interestingly, the use of jargon per se did not increase acceptance of the news source or of the content of the story. So the presence of scientific expertise is important, not so the presence of actual scientific details in the story.
  • There is no complete account of the scientific method, and again one can choose certain methods rather than others, depending on what one is trying to accomplish (a choice that is itself informed by one’s values). And of course the Duhem-Quine thesis shows that there is no straightforward way to falsify scientific theories (contra Popper). If there were supernatural causes that interact with (or override) the causes being studied by science, but are themselves undiscoverable, this would lead to false conclusions and bad predictions. Which means that the truth is discoverable empirically only if such supernatural causes are not active. Science cannot answer the question of whether such factors are present, which raises the question of whether we ought to proceed as if they were not (i.e., methodological naturalism).
  •  
    Expertise is often thought of in terms of skills, but within the context of science communication it really refers to authority and credibility. Expertise is communicated at least in part through the use of jargon, with which of course most journalists are not familiar. Jargon provides an air of authority, but at the same time the concepts referred to become inaccessible to non-specialists. Interestingly, journalists prefer sources that limit the use of jargon, but they themselves deploy jargon to demonstrate scientific proficiency.
Weiye Loh

On the Media: Survey shows that not all polls are equal - latimes.com - 0 views

  • Internet surveys sometimes acknowledge how unscientific (read: meaningless) they really are. They surely must be a pale imitation of the rigorous, carefully sampled, thoroughly transparent polls favored by political savants and mainstream news organizations
  • The line between junk and credible polling remains. But it became a little blurrier — creating concern among professional survey organizations and reason for greater skepticism by all of us — because of charges this week that one widely cited pollster may have fabricated data or manipulated it so seriously as to render it meaningless.
  • founder of the left-leaning Daily Kos website, filed a lawsuit in federal court in Oakland on Wednesday charging that Research 2000, the organization he had commissioned for 1 1/2 years to test voter opinion, had doctored its results.
  • ...4 more annotations...
  • The firm's protestations that it did nothing wrong have been loud and repeated. Evidence against the company is somewhat arcane. Suffice it to say that independent statisticians have found a bewildering lack of statistical "noise" in the company's data. Where random variation would be expected, results are too consistent.
  • Most reputable pollsters agree on one thing — polling organizations should publicly disclose as much of their methodology as possible. Just for starters, they should reveal how many people were interviewed, how they were selected, how many rejected the survey, how "likely voters" and other sub-groups were defined and how the raw data was weighted to reflect the population, or subgroups.
  • Michael Cornfield, a George Washington University political scientist and polling expert, recommends that concerned citizens ignore the lone, sometimes sensational, poll result. "Trend data are superior to a single point in time," Cornfield said via e-mail, "and consensus results from multiple firms are superior to those conducted by a single outfit."
  • The rest of us should look at none of the polls. Or look at all of them. And look out for the operators not willing to tell us how they're doing business.
  •  
    On the Media: Survey shows not all polls equal
Weiye Loh

When the scientific evidence is unwelcome, people try to reason it away | Ben Goldacre ... - 0 views

  • Each group found extensive methodological holes in the evidence they disagreed with, but ignored the very same holes in the evidence that reinforced their views.
  • Some people go even further than this when presented with unwelcome data, and decide that science itself is broken.
  •  
    When the scientific evidence is unwelcome, people try to reason it away Research results not consistent with your world view? Then you're likely to believe science can't supply all the answers
Weiye Loh

The School Issue - Junior High - Coming Out in Middle School - NYTimes.com - 1 views

  • All of this fluidity, confusion and experimentation can be understandably disorienting for parents and educators. Is an eighth grader who says he’s gay just experimenting? Could he change his mind in a week, as 13-year-olds routinely do with other identities — skater, prep, goth, jock — they try on for a while and then shed for another? And if sexuality is so fluid, should he really box himself in with a gay identity? Many parents told me they especially struggled with that last question.
    • Weiye Loh
       
      Could this fluidity be a methodology for survival? A play of signs and seduction? In what way could it be informed by the new media? I am reminded of Peter Steiner's dog-on-internet comic.
  • A year earlier they asked Austin if he was gay after they discovered his call to a gay chat line. He promised them that he was straight, and he promised himself that he would cover his tracks better. It’s not uncommon for gay youth to have their same-sex attraction discovered thanks to a rogue number on a phone bill or, more often these days, a poorly concealed Internet search history. “We see a lot of kids get outed by porn on the computer,” Tim Gillean told me in Tulsa. “I knew one kid who told his mom: ‘I don’t know how that got there. Maybe it was dad!’ ”
    • Weiye Loh
       
      Issue of privacy and surveillance made possible by technology.
Weiye Loh

New voting methods and fair elections : The New Yorker - 0 views

  • history of voting math comes mainly in two chunks: the period of the French Revolution, when some members of France’s Academy of Sciences tried to deduce a rational way of conducting elections, and the nineteen-fifties onward, when economists and game theorists set out to show that this was impossible
  • The first mathematical account of vote-splitting was given by Jean-Charles de Borda, a French mathematician and a naval hero of the American Revolutionary War. Borda concocted examples in which one knows the order in which each voter would rank the candidates in an election, and then showed how easily the will of the majority could be frustrated in an ordinary vote. Borda’s main suggestion was to require voters to rank candidates, rather than just choose one favorite, so that a winner could be calculated by counting points awarded according to the rankings. The key idea was to find a way of taking lower preferences, as well as first preferences, into account.Unfortunately, this method may fail to elect the majority’s favorite—it could, in theory, elect someone who was nobody’s favorite. It is also easy to manipulate by strategic voting.
  • If the candidate who is your second preference is a strong challenger to your first preference, you may be able to help your favorite by putting the challenger last. Borda’s response was to say that his system was intended only for honest men.
  • ...15 more annotations...
  • After the Academy dropped Borda’s method, it plumped for a simple suggestion by the astronomer and mathematician Pierre-Simon Laplace, who was an important contributor to the theory of probability. Laplace’s rule insisted on an over-all majority: at least half the votes plus one. If no candidate achieved this, nobody was elected to the Academy.
  • Another early advocate of proportional representation was John Stuart Mill, who, in 1861, wrote about the critical distinction between “government of the whole people by the whole people, equally represented,” which was the ideal, and “government of the whole people by a mere majority of the people exclusively represented,” which is what winner-takes-all elections produce. (The minority that Mill was most concerned to protect was the “superior intellects and characters,” who he feared would be swamped as more citizens got the vote.)
  • The key to proportional representation is to enlarge constituencies so that more than one winner is elected in each, and then try to align the share of seats won by a party with the share of votes it receives. These days, a few small countries, including Israel and the Netherlands, treat their entire populations as single constituencies, and thereby get almost perfectly proportional representation. Some places require a party to cross a certain threshold of votes before it gets any seats, in order to filter out extremists.
  • The main criticisms of proportional representation are that it can lead to unstable coalition governments, because more parties are successful in elections, and that it can weaken the local ties between electors and their representatives. Conveniently for its critics, and for its defenders, there are so many flavors of proportional representation around the globe that you can usually find an example of whatever point you want to make. Still, more than three-quarters of the world’s rich countries seem to manage with such schemes.
  • The alternative voting method that will be put to a referendum in Britain is not proportional representation: it would elect a single winner in each constituency, and thus steer clear of what foreigners put up with. Known in the United States as instant-runoff voting, the method was developed around 1870 by William Ware
  • In instant-runoff elections, voters rank all or some of the candidates in order of preference, and votes may be transferred between candidates. The idea is that your vote may count even if your favorite loses. If any candidate gets more than half of all the first-preference votes, he or she wins, and the game is over. But, if there is no majority winner, the candidate with the fewest first-preference votes is eliminated. Then the second-preference votes of his or her supporters are distributed to the other candidates. If there is still nobody with more than half the votes, another candidate is eliminated, and the process is repeated until either someone has a majority or there are only two candidates left, in which case the one with the most votes wins. Third, fourth, and lower preferences will be redistributed if a voter’s higher preferences have already been transferred to candidates who were eliminated earlier.
  • At first glance, this is an appealing approach: it is guaranteed to produce a clear winner, and more voters will have a say in the election’s outcome. Look more closely, though, and you start to see how peculiar the logic behind it is. Although more people’s votes contribute to the result, they do so in strange ways. Some people’s second, third, or even lower preferences count for as much as other people’s first preferences. If you back the loser of the first tally, then in the subsequent tallies your second (and maybe lower) preferences will be added to that candidate’s first preferences. The winner’s pile of votes may well be a jumble of first, second, and third preferences.
  • Such transferrable-vote elections can behave in topsy-turvy ways: they are what mathematicians call “non-monotonic,” which means that something can go up when it should go down, or vice versa. Whether a candidate who gets through the first round of counting will ultimately be elected may depend on which of his rivals he has to face in subsequent rounds, and some votes for a weaker challenger may do a candidate more good than a vote for that candidate himself. In short, a candidate may lose if certain voters back him, and would have won if they hadn’t. Supporters of instant-runoff voting say that the problem is much too rare to worry about in real elections, but recent work by Robert Norman, a mathematician at Dartmouth, suggests otherwise. By Norman’s calculations, it would happen in one in five close contests among three candidates who each have between twenty-five and forty per cent of first-preference votes. With larger numbers of candidates, it would happen even more often. It’s rarely possible to tell whether past instant-runoff elections have gone topsy-turvy in this way, because full ballot data aren’t usually published. But, in Burlington’s 2006 and 2009 mayoral elections, the data were published, and the 2009 election did go topsy-turvy.
  • Kenneth Arrow, an economist at Stanford, examined a set of requirements that you’d think any reasonable voting system could satisfy, and proved that nothing can meet them all when there are more than two candidates. So designing elections is always a matter of choosing a lesser evil. When the Royal Swedish Academy of Sciences awarded Arrow a Nobel Prize, in 1972, it called his result “a rather discouraging one, as regards the dream of a perfect democracy.” Szpiro goes so far as to write that “the democratic world would never be the same again,
  • There is something of a loophole in Arrow’s demonstration. His proof applies only when voters rank candidates; it would not apply if, instead, they rated candidates by giving them grades. First-past-the-post voting is, in effect, a crude ranking method in which voters put one candidate in first place and everyone else last. Similarly, in the standard forms of proportional representation voters rank one party or group of candidates first, and all other parties and candidates last. With rating methods, on the other hand, voters would give all or some candidates a score, to say how much they like them. They would not have to say which is their favorite—though they could in effect do so, by giving only him or her their highest score—and they would not have to decide on an order of preference for the other candidates.
  • One such method is widely used on the Internet—to rate restaurants, movies, books, or other people’s comments or reviews, for example. You give numbers of stars or points to mark how much you like something. To convert this into an election method, count each candidate’s stars or points, and the winner is the one with the highest average score (or the highest total score, if voters are allowed to leave some candidates unrated). This is known as range voting, and it goes back to an idea considered by Laplace at the start of the nineteenth century. It also resembles ancient forms of acclamation in Sparta. The more you like something, the louder you bash your shield with your spear, and the biggest noise wins. A recent variant, developed by two mathematicians in Paris, Michel Balinski and Rida Laraki, uses familiar language rather than numbers for its rating scale. Voters are asked to grade each candidate as, for example, “Excellent,” “Very Good,” “Good,” “Insufficient,” or “Bad.” Judging politicians thus becomes like judging wines, except that you can drive afterward.
  • Range and approval voting deal neatly with the problem of vote-splitting: if a voter likes Nader best, and would rather have Gore than Bush, he or she can approve Nader and Gore but not Bush. Above all, their advocates say, both schemes give voters more options, and would elect the candidate with the most over-all support, rather than the one preferred by the largest minority. Both can be modified to deliver forms of proportional representation.
  • Whether such ideas can work depends on how people use them. If enough people are carelessly generous with their approval votes, for example, there could be some nasty surprises. In an unlikely set of circumstances, the candidate who is the favorite of more than half the voters could lose. Parties in an approval election might spend less time attacking their opponents, in order to pick up positive ratings from rivals’ supporters, and critics worry that it would favor bland politicians who don’t stand for anything much. Defenders insist that such a strategy would backfire in subsequent elections, if not before, and the case of Ronald Reagan suggests that broad appeal and strong views aren’t mutually exclusive.
  • Why are the effects of an unfamiliar electoral system so hard to puzzle out in advance? One reason is that political parties will change their campaign strategies, and voters the way they vote, to adapt to the new rules, and such variables put us in the realm of behavior and culture. Meanwhile, the technical debate about electoral systems generally takes place in a vacuum from which voters’ capriciousness and local circumstances have been pumped out. Although almost any alternative voting scheme now on offer is likely to be better than first past the post, it’s unrealistic to think that one voting method would work equally well for, say, the legislature of a young African republic, the Presidency of an island in Oceania, the school board of a New England town, and the assembly of a country still scarred by civil war. If winner takes all is a poor electoral system, one size fits all is a poor way to pick its replacements.
  • Mathematics can suggest what approaches are worth trying, but it can’t reveal what will suit a particular place, and best deliver what we want from a democratic voting system: to create a government that feels legitimate to people—to reconcile people to being governed, and give them reason to feel that, win or lose (especially lose), the game is fair.
  •  
    WIN OR LOSE No voting system is flawless. But some are less democratic than others. by Anthony Gottlieb
Weiye Loh

Rational Irrationality: Do Good Kindergarten Teachers Raise their Pupils' Wages? : The ... - 0 views

  • Columnist David Leonhardt reports the findings of a new study which suggests that children who are fortunate enough to have an unusually good kindergarten teacher can expect to make roughly an extra twenty dollars a week by the age of twenty-seven.
  • it implies that during each school year a good kindergarten teacher creates an additional $320,000 of earnings.
  • The new research (pdf), the work of six economists—four from Harvard, one from Berkeley, and one from Northwestern—upends this finding. It is based on test scores and demographic data from a famous experiment carried out in Tennessee during the late nineteen-eighties, which tracked the progress of about 11,500 students from kindergarten to third grade. Most of these students are now about thirty years old, which means they have been working for up to twelve years. The researchers also gained access to income-tax data and matched it up with the test scores. Their surprising conclusion is that the uplifting effect of a good kindergarten experience, after largely disappearing during a child’s teen years, somehow reappears in the adult workplace. (See Figure 7 in the paper.) Why does this happen? The author don’t say, but Leonhardt offers this explanation: “Good early education can impart skills that last a lifetime—patience, discipline, manners, perseverance. The tests that 5-year-olds take may pick up these skills, even if later multiple-choice tests do not.”
  • ...3 more annotations...
  • However, as I read the story and the findings it is based upon, some questions crept into my mind. I relate them not out of any desire to discredit the study, which is enterprising and newsworthy, but simply as a warning to parents and policymakers not to go overboard.
  • from a dense academic article that hasn’t been published or peer reviewed. At this stage, there isn’t even a working paper detailing how the results were arrived at: just a set of slides. Why is this important? Because economics is a disputatious subject, and surprising empirical findings invariably get challenged by rival groups of researchers. The authors of the paper include two rising stars of the economics profession—Berkeley’s Emmanuel Saez and Harvard’s Raj Chetty—both of whom have reputations for careful and rigorous work. However, many other smart researchers have had their findings overturned. That is how science proceeds. Somebody says something surprising, and others in the field try to knock it down. Sometimes they succeed; sometimes they don’t. Until that Darwinian process is completed, which won’t be for another couple of years, at least, the new findings should be regarded as provisional.
  • A second point, which is related to the first, concerns methodology. In coming up with the $320,000 a year figure for the effects that kindergarten teachers have on adult earnings, the authors make use of complicated statistical techniques, including something called a “jack knife regression.” Such methods are perfectly legitimate and are now used widely in economics, but their application often adds an additional layer of ambiguity to the findings they generate. Is this particular statistical method appropriate for the task at hand? Do other methods generate different results? These are the sorts of question that other researchers will be pursuing.
  •  
    JULY 29, 2010 DO GOOD KINDERGARTEN TEACHERS RAISE THEIR PUPILS' WAGES? Posted by John Cassidy
Weiye Loh

Skepticblog » Further Thoughts on the Ethics of Skepticism - 0 views

  • My recent post “The War Over ‘Nice’” (describing the blogosphere’s reaction to Phil Plait’s “Don’t Be a Dick” speech) has topped out at more than 200 comments.
  • Many readers appear to object (some strenuously) to the very ideas of discussing best practices, seeking evidence of efficacy for skeptical outreach, matching strategies to goals, or encouraging some methods over others. Some seem to express anger that a discussion of best practices would be attempted at all. 
  • No Right or Wrong Way? The milder forms of these objections run along these lines: “Everyone should do their own thing.” “Skepticism needs all kinds of approaches.” “There’s no right or wrong way to do skepticism.” “Why are we wasting time on these abstract meta-conversations?”
  • ...12 more annotations...
  • More critical, in my opinion, is the implication that skeptical research and communication happens in an ethical vacuum. That just isn’t true. Indeed, it is dangerous for a field which promotes and attacks medical treatments, accuses people of crimes, opines about law enforcement practices, offers consumer advice, and undertakes educational projects to pretend that it is free from ethical implications — or obligations.
  • there is no monolithic “one true way to do skepticism.” No, the skeptical world does not break down to nice skeptics who get everything right, and mean skeptics who get everything wrong. (I’m reminded of a quote: “If only there were evil people somewhere insidiously committing evil deeds, and it were necessary only to separate them from the rest of us and destroy them. But the line dividing good and evil cuts through the heart of every human being.”) No one has all the answers. Certainly I don’t, and neither does Phil Plait. Nor has anyone actually proposed a uniform, lockstep approach to skepticism. (No one has any ability to enforce such a thing, in any event.)
  • However, none of that implies that all approaches to skepticism are equally valid, useful, or good. As in other fields, various skeptical practices do more or less good, cause greater or lesser harm, or generate various combinations of both at the same time. For that reason, skeptics should strive to find ways to talk seriously about the practices and the ethics of our field. Skepticism has blossomed into something that touches a lot of lives — and yet it is an emerging field, only starting to come into its potential. We need to be able to talk about that potential, and about the pitfalls too.
  • All of the fields from which skepticism borrows (such as medicine, education, psychology, journalism, history, and even arts like stage magic and graphic design) have their own standards of professional ethics. In some cases those ethics are well-explored professional fields in their own right (consider medical ethics, a field with its own academic journals and doctoral programs). In other cases those ethical guidelines are contested, informal, vague, or honored more in the breach. But in every case, there are serious conversations about the ethical implications of professional practice, because those practices impact people’s lives. Why would skepticism be any different?
  • , Skeptrack speaker Barbara Drescher (a cognitive pyschologist who teaches research methodology) described the complexity of research ethics in her own field. Imagine, she said, that a psychologist were to ask research subjects a question like, “Do your parents like the color red?” Asking this may seem trivial and harmless, but it is nonetheless an ethical trade-off with associated risks (however small) that psychological researchers are ethically obliged to confront. What harm might that question cause if a research subject suffers from erythrophobia, or has a sick parent — or saw their parents stabbed to death?
  • When skeptics undertake scientific, historical, or journalistic research, we should (I argue) consider ourselves bound by some sort of research ethics. For now, we’ll ignore the deeper, detailed question of what exactly that looks like in practical terms (when can skeptics go undercover or lie to get information? how much research does due diligence require? and so on). I’d ask only that we agree on the principle that skeptical research is not an ethical free-for-all.
  • when skeptics communicate with the public, we take on further ethical responsibilities — as do doctors, journalists, and teachers. We all accept that doctors are obliged to follow some sort of ethical code, not only of due diligence and standard of care, but also in their confidentiality, manner, and the factual information they disclose to patients. A sentence that communicates a diagnosis, prescription, or piece of medical advice (“you have cancer” or “undertake this treatment”) is not a contextless statement, but a weighty, risky, ethically serious undertaking that affects people’s lives. It matters what doctors say, and it matters how they say it.
  • Grassroots Ethics It happens that skepticism is my professional field. It’s natural that I should feel bound by the central concerns of that field. How can we gain reliable knowledge about weird things? How can we communicate that knowledge effectively? And, how can we pursue that practice ethically?
  • At the same time, most active skeptics are not professionals. To what extent should grassroots skeptics feel obligated to consider the ethics of skeptical activism? Consider my own status as a medical amateur. I almost need super-caps-lock to explain how much I am not a doctor. My medical training began and ended with a couple First Aid courses (and those way back in the day). But during those short courses, the instructors drummed into us the ethical considerations of our minimal training. When are we obligated to perform first aid? When are we ethically barred from giving aid? What if the injured party is unconscious or delirious? What if we accidentally kill or injure someone in our effort to give aid? Should we risk exposure to blood-borne illnesses? And so on. In a medical context, ethics are determined less by professional status, and more by the harm we can cause or prevent by our actions.
  • police officers are barred from perjury, and journalists from libel — and so are the lay public. We expect schoolteachers not to discuss age-inappropriate topics with our young children, or to persuade our children to adopt their religion; when we babysit for a neighbor, we consider ourselves bound by similar rules. I would argue that grassroots skeptics take on an ethical burden as soon as they speak out on medical matters, legal matters, or other matters of fact, whether from platforms as large as network television, or as small as a dinner party. The size of that burden must depend somewhat on the scale of the risks: the number of people reached, the certainty expressed, the topics tackled.
  • tu-quoque argument.
  • How much time are skeptics going to waste, arguing in a circular firing squad about each other’s free speech? Like it or not, there will always be confrontational people. You aren’t going to get a group of people as varied as skeptics are, and make them all agree to “be nice”. It’s a pipe dream, and a waste of time.
  •  
    FURTHER THOUGHTS ON THE ETHICS OF SKEPTICISM
Weiye Loh

Android software piracy rampant despite Google's efforts to curb - Computerworld - 0 views

  • Some have argued that piracy is rampant in those countries where the online Android Market is not yet available. But a recent KeyesLabs research project suggests that may not be true. KeyesLabs created a rough methodology to track total downloads of its apps, determine which ones were pirated, and the location of the end users. The results were posted in August, along with a “heat map” showing pirate activity. 
  • In July 2010, Google announced the Google Licensing Service, available via Android Market. Applications can include the new License Verification Library (LVL). “At run time, with the inclusion of a set of libraries provided by us, your application can query the Android Market licensing server to determine the license status of your users,” according to a blog post by Android engineer Eric Chu. “It returns information on whether your users are authorized to use the app based on stored sales records.”
  • Justin Case, at the Android Police Web site, dissected the LVL. “A minor patch to an application employing this official, Google-recommended protection system will render it completely worthless,” he concluded.
  • ...2 more annotations...
  • In response, Google has promised continued improvements and outlined a multipronged strategy around the new licensing service to make piracy much harder. “A determined attacker who’s willing to disassemble and reassemble code can eventually hack around the service,” acknowledged Android engineer Trevor Johns in a recent blog post.  But developers can make their work much harder by combining a cluster of techniques, he counsels: obfuscating code, modifying the licensing library to protect against common cracking techniques, designing the app to be tamper-resistant, and offloading license validation to a trusted server.
  • Gareau isn’t quite as convinced of the benefits of code obfuscation, though he does make use of it. He’s taken several other steps to protect his software work. One is providing a free trial version, which allows only a limited amount of data but is otherwise fully-featured. The idea: Let customers prove that the app will do everything they want, and they may be more willing to pay for it. He also provides a way to detect whether the app has been tampered with, for example, by removing the licensing checks. If yes, the app can be structured to stop working or behave erratically.
  •  
    Android software piracy rampant despite Google's efforts to curb
Weiye Loh

Odds Are, It's Wrong - Science News - 0 views

  • science has long been married to mathematics. Generally it has been for the better. Especially since the days of Galileo and Newton, math has nurtured science. Rigorous mathematical methods have secured science’s fidelity to fact and conferred a timeless reliability to its findings.
  • a mutant form of math has deflected science’s heart from the modes of calculation that had long served so faithfully. Science was seduced by statistics, the math rooted in the same principles that guarantee profits for Las Vegas casinos. Supposedly, the proper use of statistics makes relying on scientific results a safe bet. But in practice, widespread misuse of statistical methods makes science more like a crapshoot.
  • science’s dirtiest secret: The “scientific method” of testing hypotheses by statistical analysis stands on a flimsy foundation. Statistical tests are supposed to guide scientists in judging whether an experimental result reflects some real effect or is merely a random fluke, but the standard methods mix mutually inconsistent philosophies and offer no meaningful basis for making such decisions. Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted. As a result, countless conclusions in the scientific literature are erroneous, and tests of medical dangers or treatments are often contradictory and confusing.
  • ...24 more annotations...
  • Experts in the math of probability and statistics are well aware of these problems and have for decades expressed concern about them in major journals. Over the years, hundreds of published papers have warned that science’s love affair with statistics has spawned countless illegitimate findings. In fact, if you believe what you read in the scientific literature, you shouldn’t believe what you read in the scientific literature.
  • “There are more false claims made in the medical literature than anybody appreciates,” he says. “There’s no question about that.”Nobody contends that all of science is wrong, or that it hasn’t compiled an impressive array of truths about the natural world. Still, any single scientific study alone is quite likely to be incorrect, thanks largely to the fact that the standard statistical system for drawing conclusions is, in essence, illogical. “A lot of scientists don’t understand statistics,” says Goodman. “And they don’t understand statistics because the statistics don’t make sense.”
  • In 2007, for instance, researchers combing the medical literature found numerous studies linking a total of 85 genetic variants in 70 different genes to acute coronary syndrome, a cluster of heart problems. When the researchers compared genetic tests of 811 patients that had the syndrome with a group of 650 (matched for sex and age) that didn’t, only one of the suspect gene variants turned up substantially more often in those with the syndrome — a number to be expected by chance.“Our null results provide no support for the hypothesis that any of the 85 genetic variants tested is a susceptibility factor” for the syndrome, the researchers reported in the Journal of the American Medical Association.How could so many studies be wrong? Because their conclusions relied on “statistical significance,” a concept at the heart of the mathematical analysis of modern scientific experiments.
  • Statistical significance is a phrase that every science graduate student learns, but few comprehend. While its origins stretch back at least to the 19th century, the modern notion was pioneered by the mathematician Ronald A. Fisher in the 1920s. His original interest was agriculture. He sought a test of whether variation in crop yields was due to some specific intervention (say, fertilizer) or merely reflected random factors beyond experimental control.Fisher first assumed that fertilizer caused no difference — the “no effect” or “null” hypothesis. He then calculated a number called the P value, the probability that an observed yield in a fertilized field would occur if fertilizer had no real effect. If P is less than .05 — meaning the chance of a fluke is less than 5 percent — the result should be declared “statistically significant,” Fisher arbitrarily declared, and the no effect hypothesis should be rejected, supposedly confirming that fertilizer works.Fisher’s P value eventually became the ultimate arbiter of credibility for science results of all sorts
  • But in fact, there’s no logical basis for using a P value from a single study to draw any conclusion. If the chance of a fluke is less than 5 percent, two possible conclusions remain: There is a real effect, or the result is an improbable fluke. Fisher’s method offers no way to know which is which. On the other hand, if a study finds no statistically significant effect, that doesn’t prove anything, either. Perhaps the effect doesn’t exist, or maybe the statistical test wasn’t powerful enough to detect a small but real effect.
  • Soon after Fisher established his system of statistical significance, it was attacked by other mathematicians, notably Egon Pearson and Jerzy Neyman. Rather than testing a null hypothesis, they argued, it made more sense to test competing hypotheses against one another. That approach also produces a P value, which is used to gauge the likelihood of a “false positive” — concluding an effect is real when it actually isn’t. What  eventually emerged was a hybrid mix of the mutually inconsistent Fisher and Neyman-Pearson approaches, which has rendered interpretations of standard statistics muddled at best and simply erroneous at worst. As a result, most scientists are confused about the meaning of a P value or how to interpret it. “It’s almost never, ever, ever stated correctly, what it means,” says Goodman.
  • experimental data yielding a P value of .05 means that there is only a 5 percent chance of obtaining the observed (or more extreme) result if no real effect exists (that is, if the no-difference hypothesis is correct). But many explanations mangle the subtleties in that definition. A recent popular book on issues involving science, for example, states a commonly held misperception about the meaning of statistical significance at the .05 level: “This means that it is 95 percent certain that the observed difference between groups, or sets of samples, is real and could not have arisen by chance.”
  • That interpretation commits an egregious logical error (technical term: “transposed conditional”): confusing the odds of getting a result (if a hypothesis is true) with the odds favoring the hypothesis if you observe that result. A well-fed dog may seldom bark, but observing the rare bark does not imply that the dog is hungry. A dog may bark 5 percent of the time even if it is well-fed all of the time. (See Box 2)
    • Weiye Loh
       
      Does the problem then, lie not in statistics, but the interpretation of statistics? Is the fallacy of appeal to probability is at work in such interpretation? 
  • Another common error equates statistical significance to “significance” in the ordinary use of the word. Because of the way statistical formulas work, a study with a very large sample can detect “statistical significance” for a small effect that is meaningless in practical terms. A new drug may be statistically better than an old drug, but for every thousand people you treat you might get just one or two additional cures — not clinically significant. Similarly, when studies claim that a chemical causes a “significantly increased risk of cancer,” they often mean that it is just statistically significant, possibly posing only a tiny absolute increase in risk.
  • Statisticians perpetually caution against mistaking statistical significance for practical importance, but scientific papers commit that error often. Ziliak studied journals from various fields — psychology, medicine and economics among others — and reported frequent disregard for the distinction.
  • “I found that eight or nine of every 10 articles published in the leading journals make the fatal substitution” of equating statistical significance to importance, he said in an interview. Ziliak’s data are documented in the 2008 book The Cult of Statistical Significance, coauthored with Deirdre McCloskey of the University of Illinois at Chicago.
  • Multiplicity of mistakesEven when “significance” is properly defined and P values are carefully calculated, statistical inference is plagued by many other problems. Chief among them is the “multiplicity” issue — the testing of many hypotheses simultaneously. When several drugs are tested at once, or a single drug is tested on several groups, chances of getting a statistically significant but false result rise rapidly.
  • Recognizing these problems, some researchers now calculate a “false discovery rate” to warn of flukes disguised as real effects. And genetics researchers have begun using “genome-wide association studies” that attempt to ameliorate the multiplicity issue (SN: 6/21/08, p. 20).
  • Many researchers now also commonly report results with confidence intervals, similar to the margins of error reported in opinion polls. Such intervals, usually given as a range that should include the actual value with 95 percent confidence, do convey a better sense of how precise a finding is. But the 95 percent confidence calculation is based on the same math as the .05 P value and so still shares some of its problems.
  • Statistical problems also afflict the “gold standard” for medical research, the randomized, controlled clinical trials that test drugs for their ability to cure or their power to harm. Such trials assign patients at random to receive either the substance being tested or a placebo, typically a sugar pill; random selection supposedly guarantees that patients’ personal characteristics won’t bias the choice of who gets the actual treatment. But in practice, selection biases may still occur, Vance Berger and Sherri Weinstein noted in 2004 in ControlledClinical Trials. “Some of the benefits ascribed to randomization, for example that it eliminates all selection bias, can better be described as fantasy than reality,” they wrote.
  • Randomization also should ensure that unknown differences among individuals are mixed in roughly the same proportions in the groups being tested. But statistics do not guarantee an equal distribution any more than they prohibit 10 heads in a row when flipping a penny. With thousands of clinical trials in progress, some will not be well randomized. And DNA differs at more than a million spots in the human genetic catalog, so even in a single trial differences may not be evenly mixed. In a sufficiently large trial, unrandomized factors may balance out, if some have positive effects and some are negative. (See Box 3) Still, trial results are reported as averages that may obscure individual differences, masking beneficial or harm­ful effects and possibly leading to approval of drugs that are deadly for some and denial of effective treatment to others.
  • nother concern is the common strategy of combining results from many trials into a single “meta-analysis,” a study of studies. In a single trial with relatively few participants, statistical tests may not detect small but real and possibly important effects. In principle, combining smaller studies to create a larger sample would allow the tests to detect such small effects. But statistical techniques for doing so are valid only if certain criteria are met. For one thing, all the studies conducted on the drug must be included — published and unpublished. And all the studies should have been performed in a similar way, using the same protocols, definitions, types of patients and doses. When combining studies with differences, it is necessary first to show that those differences would not affect the analysis, Goodman notes, but that seldom happens. “That’s not a formal part of most meta-analyses,” he says.
  • Meta-analyses have produced many controversial conclusions. Common claims that antidepressants work no better than placebos, for example, are based on meta-analyses that do not conform to the criteria that would confer validity. Similar problems afflicted a 2007 meta-analysis, published in the New England Journal of Medicine, that attributed increased heart attack risk to the diabetes drug Avandia. Raw data from the combined trials showed that only 55 people in 10,000 had heart attacks when using Avandia, compared with 59 people per 10,000 in comparison groups. But after a series of statistical manipulations, Avandia appeared to confer an increased risk.
  • combining small studies in a meta-analysis is not a good substitute for a single trial sufficiently large to test a given question. “Meta-analyses can reduce the role of chance in the interpretation but may introduce bias and confounding,” Hennekens and DeMets write in the Dec. 2 Journal of the American Medical Association. “Such results should be considered more as hypothesis formulating than as hypothesis testing.”
  • Some studies show dramatic effects that don’t require sophisticated statistics to interpret. If the P value is 0.0001 — a hundredth of a percent chance of a fluke — that is strong evidence, Goodman points out. Besides, most well-accepted science is based not on any single study, but on studies that have been confirmed by repetition. Any one result may be likely to be wrong, but confidence rises quickly if that result is independently replicated.“Replication is vital,” says statistician Juliet Shaffer, a lecturer emeritus at the University of California, Berkeley. And in medicine, she says, the need for replication is widely recognized. “But in the social sciences and behavioral sciences, replication is not common,” she noted in San Diego in February at the annual meeting of the American Association for the Advancement of Science. “This is a sad situation.”
  • Most critics of standard statistics advocate the Bayesian approach to statistical reasoning, a methodology that derives from a theorem credited to Bayes, an 18th century English clergyman. His approach uses similar math, but requires the added twist of a “prior probability” — in essence, an informed guess about the expected probability of something in advance of the study. Often this prior probability is more than a mere guess — it could be based, for instance, on previous studies.
  • it basically just reflects the need to include previous knowledge when drawing conclusions from new observations. To infer the odds that a barking dog is hungry, for instance, it is not enough to know how often the dog barks when well-fed. You also need to know how often it eats — in order to calculate the prior probability of being hungry. Bayesian math combines a prior probability with observed data to produce an estimate of the likelihood of the hunger hypothesis. “A scientific hypothesis cannot be properly assessed solely by reference to the observational data,” but only by viewing the data in light of prior belief in the hypothesis, wrote George Diamond and Sanjay Kaul of UCLA’s School of Medicine in 2004 in the Journal of the American College of Cardiology. “Bayes’ theorem is ... a logically consistent, mathematically valid, and intuitive way to draw inferences about the hypothesis.” (See Box 4)
  • In many real-life contexts, Bayesian methods do produce the best answers to important questions. In medical diagnoses, for instance, the likelihood that a test for a disease is correct depends on the prevalence of the disease in the population, a factor that Bayesian math would take into account.
  • But Bayesian methods introduce a confusion into the actual meaning of the mathematical concept of “probability” in the real world. Standard or “frequentist” statistics treat probabilities as objective realities; Bayesians treat probabilities as “degrees of belief” based in part on a personal assessment or subjective decision about what to include in the calculation. That’s a tough placebo to swallow for scientists wedded to the “objective” ideal of standard statistics. “Subjective prior beliefs are anathema to the frequentist, who relies instead on a series of ad hoc algorithms that maintain the facade of scientific objectivity,” Diamond and Kaul wrote.Conflict between frequentists and Bayesians has been ongoing for two centuries. So science’s marriage to mathematics seems to entail some irreconcilable differences. Whether the future holds a fruitful reconciliation or an ugly separation may depend on forging a shared understanding of probability.“What does probability mean in real life?” the statistician David Salsburg asked in his 2001 book The Lady Tasting Tea. “This problem is still unsolved, and ... if it remains un­solved, the whole of the statistical approach to science may come crashing down from the weight of its own inconsistencies.”
  •  
    Odds Are, It's Wrong Science fails to face the shortcomings of statistics
Weiye Loh

Breakthrough Europe: Towards a Social Theory of Climate Change - 0 views

  • Lever-Tracy confronted sociologists head on about their worrisome silence on the issue. Why have sociologists failed to address the greatest and most overwhelming challenge facing modern society? Why have the figureheads of the discipline, such as Anthony Giddens and Ulrich Beck, so far refused to apply their seminal notions of structuration and the risk society to the issue?
  • Earlier, we re-published an important contribution by Ulrich Beck, the world-renowned German sociologist and a Breakthrough Senior Fellow. More recently, Current Sociology published a powerful response by Reiner Grundmann of Aston University and Nico Stehr of Zeppelin University.
  • sociologists should not rush into the discursive arena without asking some critical questions in advance, questions such as: What exactly could sociology contribute to the debate? And, is there something we urgently need that is not addressed by other disciplines or by political proposals?
  • ...12 more annotations...
  • he authors disagree with Lever-Tracy's observation that the lack of interest in climate change among sociologists is driven by a widespread suspicion of naturalistic explanations, teleological arguments and environmental determinism.
  • While conceding that Lever-Tracy's observation may be partially true, the authors argue that more important processes are at play, including cautiousness on the part of sociologists to step into a heavily politicized debate; methodological differences with the natural sciences; and sensitivity about locating climate change in the longue durée.
  • Secondly, while Lever-Tracy argues that "natural and social change are now in lockstep with each other, operating on the same scales," and that therefore a multidisciplinary approach is needed, Grundmann and Stehr suggest that the true challenge is interdisciplinarity, as opposed to multidisciplinarity.
  • Thirdly, and this possibly the most striking observation of the article, Grundmann and Stehr challenge Lever-Tracy's argument that natural scientists have successfully made the case for anthropogenic climate change, and that therefore social scientists should cease to endlessly question this scientific consensus on the basis of a skeptical postmodern 'deconstructionism'.
  • As opposed to both Lever-Tracy's positivist view and the radical postmodern deconstructionist view, Grundmann and Stehr take the social constructivist view, which argues that that every idea is socially constructed and therefore the product of human interpretation and communication. This raises the 'intractable' specters of discourse and framing, to which we will return in a second.
  • Finally, Lever-Tracy holds that climate change needs to be posited "firmly at the heart of the discipline." Grundmann and Stehr, however, emphasize that "if this is going to [be] more than wishful thinking, we need to carefully consider the prospects of such an enterprise."
  • The importance of framing climate change in a way that allows it to resonate with the concerns of the average citizen is an issue that the Breakthrough Institute has long emphasized. Especially the apocalyptic politics of fear that is often associated with climate change tends to have a counterproductive effect on public opinion. Realizing this, Grundmann and Stehr make an important warning to sociologists: "the inherent alarmism in many social science contributions on climate change merely repeats the central message provided by mainstream media." In other words, it fails to provide the kind of distantiated observation needed to approach the issue with at least a mild degree of objectivity or impartiality.
  • While this tension is symptomatic of many social scientific attempts to get involved, we propose to study these very underlying assumptions. For example, we should ask: Does the dramatization of events lead to effective political responses? Do we need a politics of fear? Is scientific consensus instrumental for sound policies? And more generally, what are the relations between a changing technological infrastructure, social shifts and belief systems? What contribution can bottom-up initiatives have in fighting climate change? What roles are there for markets, hierarchies and voluntary action? How was it possible that the 'fight against climate change' rose from a marginal discourse to a hegemonic one (from heresy to dogma)? And will the discourse remain hegemonic or will too much pub¬lic debate about climate change lead to 'climate change fatigue'?
  • In this respect, Grundmann and Stehr make another crucial observation: "the severity of a problem does not mean that we as sociologists should forget about our analytical apparatus." Bringing the analytical apparatus of sociology back in, the hunting season for positivist approaches to knowledge and nature is opened. Grundmann and Stehr consequently criticize not only Lever-Tracy's unspoken adherence to a positivist nature-society duality, taking instead a more dialectical Marxian approach to the relationship between man and his environment, but they also criticize her idea that incremental increases in our scientific knowledge of climate change and its impacts will automatically coalesce into successful and meaningful policy responses.
  • Political decisions about climate change are made on the basis of scientific research and a host of other (economic, political, cultural) considerations. Regarding the scientific dimension, it is a common perception (one that Lever-Tracy seems to share) that the more knowledge we have, the better the political response will be. This is the assumption of the linear model of policy-making that has been dominant in the past but debunked time and again (Godin, 2006). What we increasingly realize is that knowl¬edge creation leads to an excess of information and 'objectivity' (Sarewitz, 2000). Even the consensual mechanisms of the IPCC lead to an increase in options because knowledge about climate change increases.
  • Instead, Grundmann and Stehr propose to look carefully at how we frame climate change socially and whether the hegemonic climate discourse is actually contributing to successful political action or hampering it. Defending this social constructivist approach from the unfounded allegation that it would play into the hands of the climate skeptics, the authors note that defining climate change as a social construction ... is not to diminish its importance, relevance, or reality. It simply means that sociologists study the process whereby something (like anthropogenic climate change) is transformed from a conjecture into an accepted fact. With regard to policy, we observe a near exclusive focus on carbon dioxide emissions. This framing has proven counter productive, as the Hartwell paper and other sources demonstrate (see Eastin et al., 2010; Prins et al., 2010). Reducing carbon emissions in the short term is among the most difficult tasks. More progress could be made by a re-framing of the issue, not as an issue of human sinfulness, but of human dignity. [emphasis added]
  • These observations allow the authors to come full circle, arriving right back at their first observation about the real reasons why sociologists have so far kept silent on climate change. Somehow, "there seems to be the curious conviction that lest you want to be accused of helping the fossil fuel lobbies and the climate skeptics, you better keep quiet."
  •  
    Towards a Social Theory of Climate Change
Weiye Loh

Studying the politics of online science « through the looking glass - 0 views

  • Mendick, H. and Moreau, M. (2010). Monitoring the presence and representation of  women in SET occupations in UK based online media. Bradford: The UKRC.
  • Mendick and Moreau considered the representation of women on eight ‘SET’ (science, engineering and technology) websites: New Scientist, Bad Science, the Science Museum, the Natural History Museum, Neuroskeptic, Science: So What, Watt’s Up With That and RichardDawkins.net. They also monitored SET content across eight more general sites: the BBC, Channel 4, Sky, the Guardian, the Daily Mail, Wikipedia, YouTube and Twitter.
  • Their results suggest online science informational content is male dominated in that far more men than women are present. On some websites, they found no SET women. All of the 14 people in SET identified on the sampled pages of the RichardDawkins.net website were men, and so were all 29 of those mentioned on the sampled pages of the Channel 4 website (Mendick & Moreau, 2010: 11).
  • ...8 more annotations...
  • They found less hyperlinking of women’s than men’s names (Mendick & Moreau, 2010: 7). Personally, I’d have really liked some detail as to how they came up with this, and what constituted ‘hyperlinking of women’s names’ precisely. It’s potentially an interesting finding, but I can’t quite get a grip on what they are saying.
  • They also note that the women that did appear, they were often peripheral to the main story, or ‘subject to muting’ (i.e. seen but not heard). They also noted many instances where women were pictured but remain anonymous, as if there are used to illustrate a piece – for ‘ornamental’ purposes – and give the example of the wikipedia entry on scientists, which includes a picture a women as an example, but stress she is anonymous (Mendick & Moreau, 2010: 12).
  • Echoing findings of earlier research on science in the media (e.g. the Bimbo or Boffin paper), they noted that women, when represented, tended to be associated with ‘feminine’ attributes and activities, demonstrating empathy with children and animals, etc. They also noted a clustering in specific fields. For example, in the pages they’d sampled of the Guardian, they found seven mentions of women scientists compared with twenty-eight of men, and three of the these women were in a single article, about Jane Goodall (Mendick & Moreau, 2010: 12-13).
  • The women presented were often discussed in terms of appearance, personality, sexuality and personal circumstances, again echoing previous research. They also noted that women scientists, when present, tended to be younger than the men, and there was a striking lack of ethnic diversity (Mendick & Moreau, 2010: 14).
  • I’m going to be quite critical of this research. It’s not actively bad, it just seems to lack depth and precision. I suspect Mendick and Moreau were doing their best with low resources and an overly-broad brief. I also think that we are still feeling our way in terms of working out how to study online science media, and so can learn something from such a critique.
  • Problem number one: it’s a small study, and yet a ginormous topic. I’d much rather they had looked at less, but made more of it. At times I felt like I was reading a cursory glance at online science.
  • Problem number two: the methodological script seemed a bit stuck in the print era. I felt the study lacked a feel for the variety of routes people take through online science. It lacked a sense of online science’s communities and cliques, its cultures and sub-cultures, its history and its people. It lacked context. Most of all, it lacked a sense of what I think sits at the center of online communication: the link.
  • It tries to look at too much, too quickly. We’re told that of the blog entries sampled from Bad Science, three out of four of the women mentioned were associated with ‘bad science’, compared to 12 out of 27 of the men . They follow up this a note that Goldacre has appeared on television critiquing Greenfield,­ a clip of which is on his site (Mendick & Moreau, 2010: 17-18). OK, but ‘bad’ needs unpacking here, as does the gendered nature of the area Goldacre takes aim at. As for Susan Greenfield, she is a very complex character when it comes to the politics of science and gender (one I’d say it is dangerous to treat representations of simplistically). Moreover, this is a very small sample, without much feel for the broader media context the Bad Science blog works within, including not only other platforms for Ben Goldacre’s voice but comment threads, forums and a whole community of other ‘bad science bloggers’ (and their relationships with each other)
  •  
    okmark
Weiye Loh

Adventures in Flay-land: Scepticism versus Denialism - Delingpole Part II - 0 views

  • wrote a piece about James Delingpole's unfortunate appearance on the BBC program Horizon on Monday. In that piece I refered to one of his own Telegraph articles in which he criticizes renowned sceptic Dr Ben Goldacre for betraying the principles of scepticism in his regard of the climate change debate. That article turns out to be rather instructional as it highlights perfectly the difference between real scepticism and the false scepticism commonly described as denialism.
  • It appears that James has tremendous respect for Ben Goldacre, who is a qualified medical doctor and has written a best-selling book about science scepticism called Bad Science and continues to write a popular Guardian science column. Here's what Delingpole has to say about Dr Goldacre: Many of Goldacre’s campaigns I support. I like and admire what he does. But where I don’t respect him one jot is in his views on ‘Climate Change,’ for they jar so very obviously with supposed stance of determined scepticism in the face of establishment lies.
  • Scepticism is not some sort of rebellion against the establishment as Delingpole claims. It is not in itself an ideology. It is merely an approach to evaluating new information. There are varying definitions of scepticism, but Goldacre's variety goes like this: A sceptic does not support or promote any new theory until it is proven to his or her satisfaction that the new theory is the best available. Evidence is examined and accepted or discarded depending on its persuasiveness and reliability. Sceptics like Ben Goldacre have a deep appreciation for the scientific method of testing a hypothesis through experimentation and are generally happy to change their minds when the evidence supports the opposing view. Sceptics are not true believers, but they search for the truth. Far from challenging the established scientific consensus, Goldacre in Bad Science typcially defends the scientific consensus against alternative medical views that fall back on untestable positions. In science the consensus is sometimes proven wrong, and while this process is imperfect it eventually results in the old consensus being replaced with a new one.
  • ...11 more annotations...
  • So the question becomes "what is denialism?" Denialism is a mindset that chooses to deny reality in order to avoid an uncomfortable truth. Denialism creates a false sense of truth through the subjective selection of evidence (cherry picking). Unhelpful evidence is rejected and excuses are made, while supporting evidence is accepted uncritically - its meaning and importance exaggerated. It is a common feature of denialism to claim the existence of some sort of powerful conspiracy to suppress the truth. Rejection by the mainstream of some piece of evidence supporting the denialist view, no matter how flawed, is taken as further proof of the supposed conspiracy. In this way the denialist always has a fallback position.
  • Delingpole makes the following claim: Whether Goldacre chooses to ignore it or not, there are many, many hugely talented, intelligent men and women out there – from mining engineer turned Hockey-Stick-breaker Steve McIntyre and economist Ross McKitrick to bloggers Donna LaFramboise and Jo Nova to physicist Richard Lindzen….and I really could go on and on – who have amassed a body of hugely powerful evidence to show that the AGW meme which has spread like a virus around the world these last 20 years is seriously flawed.
  • So he mentions a bunch of people who are intelligent and talented and have amassed evidence to the effect that the consensus of AGW (Anthropogenic Global Warming) is a myth. Should I take his word for it? No. I am a sceptic. I will examine the evidence and the people behind it.
  • MM claims that global temperatures are not accelerating. The claims have however been roundly disproved as explained here. It is worth noting at this point that neither man is a climate scientist. McKitrick is an economist and McIntyre is a mining industry policy analyst. It is clear from the very detailed rebuttal article that McIntrye and McKitrick have no qualifications to critique the earlier paper and betray fundamental misunderstandings of methodologies employed in that study.
  • This Wikipedia article explains in better laymens terms how the MM claims are faulty.
  • It is difficult for me to find out much about blogger Donna LaFrambois. As far as I can see she runs her own blog at http://nofrakkingconsensus.wordpress.com and is the founder of another site here http://www.noconsensus.org/. It's not very clear to me what her credentials are
  • She seems to be a critic of the so-called climate bible, a comprehensive report by the UN Intergovernmental Panel on Climate Change (IPCC)
  • I am familiar with some of the criticisms of this panel. Working Group 2 famously overstated the estimated rate of disappearance of the Himalayan glacier in 2007 and was forced to admit the error. Working Group 2 is a panel of biologists and sociologists whose job is to evaluate the impact of climate change. These people are not climate scientists. Their report takes for granted the scientific basis of climate change, which has been delivered by Working Group 1 (the climate scientists). The science revealed by Working Group 1 is regarded as sound (of course this is just a conspiracy, right?) At any rate, I don't know why I should pay attention to this blogger. Anyone can write a blog and anyone with money can own a domain. She may be intelligent, but I don't know anything about her and with all the millions of blogs out there I'm not convinced hers is of any special significance.
  • Richard Lindzen. Okay, there's information about this guy. He has a wiki page, which is more than I can say for the previous two. He is an atmospheric physicist and Professor of Meteorology at MIT.
  • According to Wikipedia, it would seem that Lindzen is well respected in his field and represents the 3% of the climate science community who disagree with the 97% consensus.
  • The second to last paragraph of Delingpole's article asks this: If  Goldacre really wants to stick his neck out, why doesn’t he try arguing against a rich, powerful, bullying Climate-Change establishment which includes all three British main political parties, the National Academy of Sciences, the Royal Society, the Prince of Wales, the Prime Minister, the President of the USA, the EU, the UN, most schools and universities, the BBC, most of the print media, the Australian Government, the New Zealand Government, CNBC, ABC, the New York Times, Goldman Sachs, Deutsche Bank, most of the rest of the City, the wind farm industry, all the Big Oil companies, any number of rich charitable foundations, the Church of England and so on?I hope Ben won't mind if I take this one for him (first of all, Big Oil companies? Are you serious?) The answer is a question and the question is "Where is your evidence?"
Weiye Loh

Book Review: Future Babble by Dan Gardner « Critical Thinking « Skeptic North - 0 views

  • I predict that you will find this review informative. If you do, you will congratulate my foresight. If you don’t, you’ll forget I was wrong.
  • My playful intro summarizes the main thesis of Gardner’s excellent book, Future Babble: Why Expert Predictions Fail – and Why We Believe Them Anyway.
  • In Future Babble, the research area explored is the validity of expert predictions, and the primary researcher examined is Philip Tetlock. In the early 1980s, Tetlock set out to better understand the accuracy of predictions made by experts by conducting a methodologically sound large-scale experiment.
  • ...10 more annotations...
  • Gardner presents Tetlock’s experimental design in an excellent way, making it accessible to the lay person. Concisely, Tetlock examined 27450 judgments in which 284 experts were presented with clear questions whose answers could later be shown to be true or false (e.g., “Will the official unemployment rate be higher, lower or the same a year from now?”). For each prediction, the expert must answer clearly and express their degree of certainty as a percentage (e.g., dead certain = 100%). The usage of precise numbers adds increased statistical options and removes the complications of vague or ambiguous language.
  • Tetlock found the surprising and disturbing truth “that experts’ predictions were no more accurate than random guesses.” (p. 26) An important caveat is that there was a wide range of capability, with some experts being completely out of touch, and others able to make successful predictions.
  • What distinguishes the impressive few from the borderline delusional is not whether they’re liberal or conservative. Tetlock’s data showed political beliefs made no difference to an expert’s accuracy. The same is true of optimists and pessimists. It also made no difference if experts had a doctorate, extensive experience, or access to classified information. Nor did it make a difference if experts were political scientists, historians, journalists, or economists.” (p. 26)
  • The experts who did poorly were not comfortable with complexity and uncertainty, and tended to reduce most problems to some core theoretical theme. It was as if they saw the world through one lens or had one big idea that everything else had to fit into. Alternatively, the experts who did decently were self-critical, used multiple sources of information and were more comfortable with uncertainty and correcting their errors. Their thinking style almost results in a paradox: “The experts who were more accurate than others tended to be less confident they were right.” (p.27)
  • Gardner then introduces the terms ‘Hedgehog’ and ‘Fox’ to refer to bad and good predictors respectively. Hedgehogs are the ones you see pushing the same idea, while Foxes are likely in the background questioning the ability of prediction itself while making cautious proposals. Foxes are more likely to be correct. Unfortunately, it is Hedgehogs that we see on the news.
  • one of Tetlock’s findings was that “the bigger the media profile of an expert, the less accurate his predictions.” (p.28)
  • Chapter 2 – The Unpredictable World An exploration into how many events in the world are simply unpredictable. Gardner discusses chaos theory and necessary and sufficient conditions for events to occur. He supports the idea of actually saying “I don’t know,” which many experts are reluctant to do.
  • Chapter 3 – In the Minds of Experts A more detailed examination of Hedgehogs and Foxes. Gardner discusses randomness and the illusion of control while using narratives to illustrate his points à la Gladwell. This chapter provides a lot of context and background information that should be very useful to those less initiated.
  • Chapter 6 – Everyone Loves a Hedgehog More about predictions and how the media picks up hedgehog stories and talking points without much investigation into their underlying source or concern for accuracy. It is a good demolition of the absurdity of so many news “discussion shows.” Gardner demonstrates how the media prefer a show where Hedgehogs square off against each other, and it is important that these commentators not be challenged lest they become exposed and, by association, implicate the flawed structure of the program/network.Gardner really singles out certain people, like Paul Ehrlich, and shows how they have been wrong many times and yet can still get an audience.
  • “An assertion that cannot be falsified by any conceivable evidence is nothing more than dogma. It can’t be debated. It can’t be proven or disproven. It’s just something people choose to believe or not for reasons that have nothing to do with fact and logic. And dogma is what predictions become when experts and their followers go to ridiculous lengths to dismiss clear evidence that they failed.”
Weiye Loh

Roger Pielke Jr.'s Blog: Flood Disasters and Human-Caused Climate Change - 0 views

  • [UPDATE: Gavin Schmidt at Real Climate has a post on this subject that  -- surprise, surprise -- is perfectly consonant with what I write below.] [UPDATE 2: Andy Revkin has a great post on the representations of the precipitation paper discussed below by scientists and related coverage by the media.]  
  • Nature published two papers yesterday that discuss increasing precipitation trends and a 2000 flood in the UK.  I have been asked by many people whether these papers mean that we can now attribute some fraction of the global trend in disaster losses to greenhouse gas emissions, or even recent disasters such as in Pakistan and Australia.
  • I hate to pour cold water on a really good media frenzy, but the answer is "no."  Neither paper actually discusses global trends in disasters (one doesn't even discuss floods) or even individual events beyond a single flood event in the UK in 2000.  But still, can't we just connect the dots?  Isn't it just obvious?  And only deniers deny the obvious, right?
  • ...12 more annotations...
  • What seems obvious is sometime just wrong.  This of course is why we actually do research.  So why is it that we shouldn't make what seems to be an obvious connection between these papers and recent disasters, as so many have already done?
  • First, the Min et al. paper seeks to identify a GHG signal in global precipitation over the period 1950-1999.  They focus on one-day and five-day measures of precipitation.  They do not discuss streamflow or damage.  For many years, an upwards trend in precipitation has been documented, and attributed to GHGs, even back to the 1990s (I co-authored a paper on precipitation and floods in 1999 that assumed a human influence on precipitation, PDF), so I am unsure what is actually new in this paper's conclusions.
  • However, accepting that precipitation has increased and can be attributed in some part to GHG emissions, there have not been shown corresponding increases in streamflow (floods)  or damage. How can this be?  Think of it like this -- Precipitation is to flood damage as wind is to windstorm damage.  It is not enough to say that it has become windier to make a connection to increased windstorm damage -- you need to show a specific increase in those specific wind events that actually cause damage. There are a lot of days that could be windier with no increase in damage; the same goes for precipitation.
  • My understanding of the literature on streamflow is that there have not been shown increasing peak streamflow commensurate with increases in precipitation, and this is a robust finding across the literature.  For instance, one recent review concludes: Floods are of great concern in many areas of the world, with the last decade seeing major fluvial events in, for example, Asia, Europe and North America. This has focused attention on whether or not these are a result of a changing climate. Rive flows calculated from outputs from global models often suggest that high river flows will increase in a warmer, future climate. However, the future projections are not necessarily in tune with the records collected so far – the observational evidence is more ambiguous. A recent study of trends in long time series of annual maximum river flows at 195 gauging stations worldwide suggests that the majority of these flow records (70%) do not exhibit any statistically significant trends. Trends in the remaining records are almost evenly split between having a positive and a negative direction.
  • Absent an increase in peak streamflows, it is impossible to connect the dots between increasing precipitation and increasing floods.  There are of course good reasons why a linkage between increasing precipitation and peak streamflow would be difficult to make, such as the seasonality of the increase in rain or snow, the large variability of flooding and the human influence on river systems.  Those difficulties of course translate directly to a difficulty in connecting the effects of increasing GHGs to flood disasters.
  • Second, the Pall et al. paper seeks to quantify the increased risk of a specific flood event in the UK in 2000 due to greenhouse gas emissions.  It applies a methodology that was previously used with respect to the 2003 European heatwave. Taking the paper at face value, it clearly states that in England and Wales, there has not been an increasing trend in precipitation or floods.  Thus, floods in this region are not a contributor to the global increase in disaster costs.  Further, there has been no increase in Europe in normalized flood losses (PDF).  Thus, Pall et al. paper is focused attribution in the context of on a single event, and not trend detection in the region that it focuses on, much less any broader context.
  • More generally, the paper utilizes a seasonal forecast model to assess risk probabilities.  Given the performance of seasonal forecast models in actual prediction mode, I would expect many scientists to remain skeptical of this approach to attribution. Of course, if this group can show an improvement in the skill of actual seasonal forecasts by using greenhouse gas emissions as a predictor, they will have a very convincing case.  That is a high hurdle.
  • In short, the new studies are interesting and add to our knowledge.  But they do not change the state of knowledge related to trends in global disasters and how they might be related to greenhouse gases.  But even so, I expect that many will still want to connect the dots between greenhouse gas emissions and recent floods.  Connecting the dots is fun, but it is not science.
  • Jessica Weinkle said...
  • The thing about the nature articles is that Nature itself made the leap from the science findings to damages in the News piece by Q. Schiermeier through the decision to bring up the topic of insurance. (Not to mention that which is symbolically represented merely by the journal’s cover this week). With what I (maybe, naively) believe to be a particularly ballsy move, the article quoted Muir-Wood, an industry scientists. However, what he is quoted as saying is admirably clever. Initially it is stated that Dr. Muir-Wood backs the notion that one cannot put the blame of increased losses on climate change. Then, the article ends with a quote from him, “If there’s evidence that risk is changing, then this is something we need to incorporate in our models.”
  • This is a very slippery slope and a brilliant double-dog dare. Without doing anything but sitting back and watching the headlines, one can form the argument that “science” supports the remodeling of the hazard risk above the climatological average and is more important then the risks stemming from socioeconomic factors. The reinsurance industry itself has published that socioeconomic factors far outweigh changes in the hazard in concern of losses. The point is (and that which has particularly gotten my knickers in a knot) is that Nature, et al. may wish to consider what it is that they want to accomplish. Is it greater involvement of federal governments in the insurance/reinsurance industry on the premise that climate change is too great a loss risk for private industry alone regardless of the financial burden it imposes? The move of insurance mechanisms into all corners of the earth under the auspices of climate change adaptation? Or simply a move to bolster prominence, regardless of whose back it breaks- including their own, if any of them are proud owners of a home mortgage? How much faith does one have in their own model when they are told that hundreds of millions of dollars in the global economy is being bet against the odds that their models produce?
  • What Nature says matters to the world; what scientists say matters to the world- whether they care for the responsibility or not. That is after all, the game of fame and fortune (aka prestige).
Weiye Loh

Roger Pielke Jr.'s Blog: Full Comments to the Guardian - 0 views

  • The Guardian has an good article today on a threatened libel suit under UK law against Gavin Schmidt, a NASA researcher who blogs at Real Climate, by the publishers of the journal Energy and Environment. 
  • Here are my full comments to the reporter for the Guardian, who was following up on Gavin's reference to comments I had made a while back about my experiences with E&E:
  • In 2000, we published a really excellent paper (in my opinion) in E&E in that has stood the test of time: Pielke, Jr., R. A., R. Klein, and D. Sarewitz (2000), Turning the big knob: An evaluation of the use of energy policy to modulate future climate impacts. Energy and Environment 2:255-276. http://sciencepolicy.colorado.edu/admin/publication_files/resource-250-2000.07.pdf You'll see that paper was in only the second year of the journal, and we were obviously invited to submit a year or so before that. It was our expectation at the time that the journal would soon be ISI listed and it would become like any other academic journal. So why not publish in E&E?
  • ...5 more annotations...
  • That paper, like a lot of research, required a lot of effort.  So it was very disappointing to E&E in the years that followed identify itself as an outlet for alternative perspectives on the climate issue. It has published a number of low-quality papers and a high number of opinion pieces, and as far as I know it never did get ISI listed.
  • Boehmer-Christiansen's quote about following her political agenda in running the journal is one that I also have cited on numerous occasions as an example of the pathological politicization of science. In this case the editor's political agenda has clearly undermined the legitimacy of the outlet.  So if I had a time machine I'd go back and submit our paper elsewhere!
  • A consequence of the politicization of E&E is that any paper published there is subsequently ignored by the broader scientific community. In some cases perhaps that is justified, but I would argue that it provided a convenient excuse to ignore our paper on that basis alone, and not on the merits of its analysis. So the politicization of E&E enables a like response from its critics, which many have taken full advantage of. For outside observers of climate science this action and response together give the impression that scientific studies can be evaluated simply according to non-scientific criteria, which ironically undermines all of science, not just E&E.  The politicization of the peer review process is problematic regardless of who is doing the politicization because it more readily allows for political judgments to substitute for judgments of the scientific merit of specific arguments.  An irony here of course is that the East Anglia emails revealed a desire to (and some would say success in) politicize the peer review process, which I discuss in The Climate Fix.
  • For my part, in 2007 I published a follow on paper to the 2000 E&E paper that applied and extended a similar methodology.  This paper passed peer review in the Philosophical Transactions of the Royal Society: Pielke, Jr., R. A. (2007), Future economic damage from tropical cyclones: sensitivities to societal and climate changes. Philosophical Transactions of the Royal Society A 365 (1860) 2717-2729 http://sciencepolicy.colorado.edu/admin/publication_files/resource-2517-2007.14.pdf
  • Over the long run I am confident that good ideas will win out over bad ideas, but without care to the legitimacy of our science institutions -- including journals and peer review -- that long run will be a little longer.
1 - 20 of 28 Next ›
Showing 20 items per page