Group items tagged

We’re not here to defend peer review against its many critics. We have the same feelings about it that Churchill did about democracy, aka the worst form of government except for all those others that have been tried. Of course, a good number of the retractions we write about are due to misconduct, and it’s not clear how peer review, no matter how good, would detect out-and-out fraud.

With that in mind, a paper published last week in the Annals of Emergency Medicine caught our eye. Over 14 years, 84 editors at the journal rated close to 15,000 reviews by about 1,500 reviewers. Highlights of their findings: …92% of peer reviewers deteriorated during 14 years of study in the quality and usefulness of their reviews (as judged by editors at the time of decision), at rates unrelated to the length of their service (but moderately correlated with their mean quality score, with better-than average reviewers decreasing at about half the rate of those below average). Only 8% improved, and those by very small amount.

The average reviewer in our study would have taken 12.5 years to reach this threshold; only 3% of reviewers whose quality decreased would have reached it in less than 5 years, and even the worst would take 3.2 years. Another 35% of all reviewers would reach the threshold in 5 to 10 years, 28% in 10 to 15 years, 12% in 15 to 20 years, and 22% in 20 years or more. So the decline was slow. Still, the results, note the authors, were surprising: Such a negative overall trend is contrary to most editors’ and reviewers’ intuitive expectations and beliefs about reviewer skills and the benefits of experience.

t was the big news that wasn’t. Hyperbolic claims about the possible discovery of alien life, or a second branch of life on Earth, turned out to be nothing more than bacteria that can thrive on arsenic, using it in place of phosphorus in their DNA and other molecules. But after the initial layers of hype were peeled away, even this extraordinar

This is a chronological roundup of the criticism against the science in the paper itself, ending with some personal reflections on my own handling of the story (skip to Friday, December 10th for that bit).

Thursday, December 2nd: Felisa Wolfe-Simon published a paper in Science, claiming to have found bacteria in California’s Mono Lake that can grow using arsenic instead of phosphorus. Given that phosphorus is meant to be one of six irreplaceable elements, this would have been a big deal, not least because the bacteria apparently used arsenic to build the backbones of their DNA molecules.

From: Bill Hughes Cc: Sonja Boehmer-Christiansen Subject:: E&E libel Date: 02/18/11 10:48:01 Gavin, your comment about Energy & Environment which you made on RealClimate has been brought to my attention: “The evidence for this is in precisely what happens in venues like E&E that have effectively dispensed with substantive peer review for any papers that follow the editor’s political line. ” To assert, without knowing, as you cannot possibly know, not being connected with the journal yourself, that an academic journal does not bother with peer review, is a terribly damaging charge, and one I’m really quite surprised that you’re prepared to make. And to further assert that peer review is abandoned precisely in order to let the editor publish papers which support her political position, is even more damaging, not to mention being completely ridiculous. At the moment, I’m prepared to settle merely for a retraction posted on RealClimate. I’m quite happy to work with you to find a mutually satisfactory form of words: I appreciate you might find it difficult. I look forward to hearing from you. With best wishes Bill Hughes Director Multi-Science Publsihing [sic] Co Ltd

The comment in question was made in the post “From blog to Science”

The point being that if the ‘peer-review’ bar gets lowered, the result is worse submissions, less impact and a declining reputation. Something that fits E&E in spades. This conclusion is based on multiple years of evidence of shoddy peer-review at E&E and, obviously, on the statements of the editor, Sonja Boehmer-Christiansen. She was quoted by Richard Monastersky in the Chronicle of Higher Education (3 Sep 2003) in the wake of the Soon and Baliunas fiasco: The journal’s editor, Sonja Boehmer-Christiansen, a reader in geography at the University of Hull, in England, says she sometimes publishes scientific papers challenging the view that global warming is a problem, because that position is often stifled in other outlets. “I’m following my political agenda — a bit, anyway,” she says. “But isn’t that the right of the editor?”

The Templeton Foundation claims to be a friend of science. So why does it make so many researchers uneasy?

With a current endowment estimated at US$2.1 billion, the organization continues to pursue Templeton's goal of building bridges between science and religion. Each year, it doles out some $70 million in grants, more than $40 million of which goes to research in fields such as cosmology, evolutionary biology and psychology.

however, many scientists find it troubling — and some see it as a threat. Jerry Coyne, an evolutionary biologist at the University of Chicago, Illinois, calls the foundation "sneakier than the creationists". Through its grants to researchers, Coyne alleges, the foundation is trying to insinuate religious values into science. "It claims to be on the side of science, but wants to make faith a virtue," he says.

Gavin Schmidt, a climate modeller and Real Climate member based at Nasa's Goddard Institute for Space Studies in New York, has claimed that Energy & Environment (E&E) has "effectively dispensed with substantive peer review for any papers that follow the editor's political line." The journal denies the claim, and, according to Schmidt, has threatened to take further action unless he retracts it.

Every paper that is submitted to the journal is vetted by a number of experts, she said. But she did not deny that she allows her political agenda to influence which papers are published in the journal. "I'm not ashamed to say that I deliberately encourage the publication of papers that are sceptical of climate change," said Boehmer-Christiansen, who does not believe in man-made climate change.

Simon Singh, a science writer who last year won a major libel battle with the British Chiropractic Association (BCA), said: "A libel threat is potentially catastrophic. It can lead to a journalist going bankrupt or a blogger losing his house. A lot of journalists and scientists will understandably react to the threat of libel by retracting their articles, even if they are confident they are correct. So I'm delighted that Gavin Schmidt is going to stand up for what he has written." During the case with the BCA, Singh also received a libel threat in response to an article he had written about climate change, but Singh stood by what he had written and threat was not carried through.

Gartner, the research company, estimates that using communities to solve support issues can reduce costs by up to 50%. When TomTom, a maker of satellite-navigation systems, switched on social support, members handled 20,000 cases in its first two weeks and saved it around $150,000. Best Buy, an American gadget retailer, values its 600,000 users at $5m annually. 

Dear Prof. Haidt,You understandably got upset by my harsh criticism of your recent claims about the mechanisms behind the alleged anti-conservative bias that apparently so permeates the modern academy. I find it amusing that you simply assumed I had not looked at your talk and was therefore speaking without reason. Yet, I have indeed looked at it (it is currently published at Edge, a non-peer reviewed webzine), and found that it simply doesn’t add much to the substance (such as it is) of Tierney’s summary.

Yes, you do acknowledge that there may be multiple reasons for the imbalance between the number of conservative and liberal leaning academics, but then you go on to characterize the academy, at least in your field, as a tribe having a serious identity issue, with no data whatsoever to back up your preferred subset of causal explanations for the purported problem.

your talk is simply an extended op-ed piece, which starts out with a summary of your findings about the different moral outlooks of conservatives and liberals (which I have criticized elsewhere on this blog), and then proceeds to build a flimsy case based on a couple of anecdotes and some badly flawed data.

The Guardian has an good article today on a threatened libel suit under UK law against Gavin Schmidt, a NASA researcher who blogs at Real Climate, by the publishers of the journal Energy and Environment. 

Here are my full comments to the reporter for the Guardian, who was following up on Gavin's reference to comments I had made a while back about my experiences with E&E:

In 2000, we published a really excellent paper (in my opinion) in E&E in that has stood the test of time: Pielke, Jr., R. A., R. Klein, and D. Sarewitz (2000), Turning the big knob: An evaluation of the use of energy policy to modulate future climate impacts. Energy and Environment 2:255-276. http://sciencepolicy.colorado.edu/admin/publication_files/resource-250-2000.07.pdf You'll see that paper was in only the second year of the journal, and we were obviously invited to submit a year or so before that. It was our expectation at the time that the journal would soon be ISI listed and it would become like any other academic journal. So why not publish in E&E?

pharmaceutical company Merck, Sharp & Dohme paid Elsevier to produce a fake medical journal that, to any superficial examination, looked like a real medical journal but was in reality nothing more than advertising for Merck

As reported by The Scientist: Merck paid an undisclosed sum to Elsevier to produce several volumes of a publication that had the look of a peer-reviewed medical journal, but contained only reprinted or summarized articles--most of which presented data favorable to Merck products--that appeared to act solely as marketing tools with no disclosure of company sponsorship. "I've seen no shortage of creativity emanating from the marketing departments of drug companies," Peter Lurie, deputy director of the public health research group at the consumer advocacy nonprofit Public Citizen, said, after reviewing two issues of the publication obtained by The Scientist. "But even for someone as jaded as me, this is a new wrinkle." The Australasian Journal of Bone and Joint Medicine, which was published by Exerpta Medica, a division of scientific publishing juggernaut Elsevier, is not indexed in the MEDLINE database, and has no website (not even a defunct one). The Scientist obtained two issues of the journal: Volume 2, Issues 1 and 2, both dated 2003. The issues contained little in the way of advertisements apart from ads for Fosamax, a Merck drug for osteoporosis, and Vioxx.

there are numerous "throwaway" journals out there. "Throwaway" journals tend to be defined as journals that are provided free of charge, have a lot of advertising (a high "advertising-to-text" ratio, as it is often described), and contain no original investigations. Other relevant characteristics include: Supported virtually entirely by advertising revenue. Ads tend to be placed within article pages interrupting the articles, rather than between articles, as is the case with most medical journals that accept ads Virtually the entire content is reviews of existing content of variable (and often dubious) quality. Parasitic. Throwaways often summarize peer-reviewed research from real journals. Questionable (at best) peer review. Throwaways tend to cater to an uninvolved and uncritical readership. No original work.

commonly voiced criticisms of traditional peer review is that it discourages truly innovative ideas, rejecting field-changing papers while publishing ideas that fall into a status quo and the “hot” fields of the day

Another is that it is nearly impossible to immediately spot the importance of a paper—to truly evaluate a paper, one needs months, if not years, to see the impact it has on its field.

we present some papers that suggest these two criticisms are correct, at least in part. These studies were published in lower-profile journals (all with current impact factors of 6 or below), suggesting they should have had less of an impact. But these papers eventually accumulated at least 1,000 citations. Many were rejected from higher-tier journals. All changed their fields forever.

The origin of Aids denialism lies with one man. Peter Duesberg has spent the whole of his academic career at the University of California, Berkeley. In the 1970s he performed groundbreaking work that helped show how mutated genes cause cancer, an insight that earned him a well-deserved international reputation.

in the early 1980s, something changed. Duesberg attempted to refute his own theories, claiming that it was not mutated genes but rather environmental toxins that are cancer's true cause. He dismissed the studies of other researchers who had furthered his original work. Then, in 1987, he published a paper that extended his new train of thought to Aids.

Initially many scientists were open to Duesberg's ideas. But as evidence linking HIV to Aids mounted - crucially the observation that ARVs brought Aids sufferers who were on the brink of death back to life - the vast majority concluded that the debate was over. Nonetheless, Duesberg persisted with his arguments, and in doing so attracted a cabal of supporters

A group of school children aged between 8 and 10 years old have had their school science project accepted for publication in an internationally recognised peer-reviewed journal. The paper, which reports novel findings in how bumblebees perceive colour, is published in the Royal Society journal Biology Letters.

CONAN: Do you find that, though, there is a lot of ideology in this business? Prof. MULLER: Well, I think what's happened is that many scientists have gotten so concerned about global warming, correctly concerned I mean they look at it and they draw a conclusion, and then they're worried that the public has not been concerned, and so they become advocates. And at that point, it's unfortunate, I feel that they're not trusting the public. They're not presenting the science to the public. They're presenting only that aspect to the science that will convince the public. That's not the way science works. And because they don't trust the public, in the end the public doesn't trust them. And the saddest thing from this, I think, is a loss of credibility of scientists because so many of them have become advocates.

CONAN: And that's, you would say, would be at the heart of the so-called Climategate story, where emails from some scientists seemed to be working to prevent the work of other scientists from appearing in peer-reviewed journals. Prof. MULLER: That really shook me up when I learned about that. I think that Climategate is a very unfortunate thing that happened, that the scientists who were involved in that, from what I've read, didn't trust the public, didn't even trust the scientific public. They were not showing the discordant data. That's something that - as a scientist I was trained you always have to show the negative data, the data that disagrees with you, and then make the case that your case is stronger. And they were hiding the data, and a whole discussion of suppressing publications, I thought, was really unfortunate. It was not at a high point for science

Dozens of freshmen at Singapore Management University spent one evening last week learning how to "wiki," or use the software that lets large numbers of people write and edit class projects online. Though many said experiencing a public editing process similar to that of Wikipedia could prove valuable, some were wary of the collaborative tool, with its public nature and the ability to toss out or revise the work of their classmates.

It puts students in the awkward position of having to publicly correct a peer, which can cause the corrected person to lose face.

"You have to be more aware of others and have a sensitivity to others."

That's the title of The Atlantic profile of Dr. John Ioannidis who "has spent his career challenging his peers by exposing their bad science." His 2005 paper in PLoS Medicine was on why most published research findings are false.

Ioannidis anticipated that the community might shrug off his findings: sure, a lot of dubious research makes it into journals, but we researchers and physicians know to ignore it and focus on the good stuff, so what’s the big deal? The other paper headed off that claim.

He zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years, as judged by the science community’s two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals.

Every time a disaster strikes, some environmentalists blame it on climate change. “It’s been such a part of the narrative of the public and political debate, particularly after Hurricane Katrina,” Roger Pielke Jr, an expert on the politics of climate change at the University of Colorado, told me.

But nothing in the scientific literature indicates that this is true. A host of recent peer-reviewed studies agree: there’s no evidence that climate change has increased the damage from natural disasters. Most likely, climate change will make disasters worse some day, but not yet.

Laurens Bouwer, of Amsterdam’s Vrije Universiteit, has recently reviewed 22 “disaster loss studies” and concludes: “Anthropogenic climate change so far has not had a significant impact on losses from natural disasters.”

Perhaps like me, you watched the BBC Two Horizons program Monday night presented by Sir Paul Nurse, president of the Royal Society and Nobel Prize winning geneticist for his discovery of the genes of cell division.

James. He really believes there's some kind of mainstream science "warmist" conspiracy against the brave outliers who dare to challenge the consensus. He really believes that "climategate" is a real scandal. He fails to understand that it is a common practice in statistics to splice together two or more datasets where you know that the quality of data is patchy. In the case of "climategate", researchers found that indirect temperature measurements based on tree ring widths (the tree ring temperature proxy) is consistent with other proxy methods of recording temperature from before the start of the instrumental temperature record (around 1950) but begins to show a decline in temperature after that for reasons which are unclear. Actual temperature measurements however show the opposite. The researcher at the head of the climategate affair, Phil Jones, created a graph of the temperature record to include on the cover of a report for policy makers and journalists. For this graph he simply spliced together the tree ring proxy data up until 1950 with the recorded data after that using statistical techniques to bring them into agreement. What made this seem particularly dodgy was an email intercepted by a hacker in which Jones referred to this practice as a "Mike's Nature trick", referring to a paper published by his colleague Mike Hulme Michael Mann in the journal Nature. It is however nothing out of the ordinary. Delingpole and others have talked about how this "trick" was used here to "hide the decline" revealed by the other dataset, as though this was some sort of deception. The fact that all parties were found to have behaved ethically is simply further evidence of the global warmist conspiracy. Delingpole takes it further and casts aspersions on scientific consensus and the entire peer review process.

When Nurse asked Delingpole the very straightforward question of whether he would be willing to trust a scientific consensus if he required treatment for cancer, he could have said "Gee, that's an interesting question. Let me think about that and why it's different."