Group items tagged

The Intelligence Process

Intelligence means three things to us.

Reports that showcase empathetic analysis: Thinking About the Unthinkable: A U.S.-Iranian Deal Germany's Choice Hezbollah, Radical but Rational

Each week, we have a pretty good idea of where the ships will be, based on geopolitical patterns, strategies and developments. But surprises -- and they do happen -- allow us to challenge and re-evaluate our positions.

The map contains only publicly available, open-source information. We're not publishing any secrets; we're compiling available information and applying it to an easy-to-use, actionable graphic for our analysts, subscribers and clients.

First, it would be an absurd amount of work

Second, from the editor’s perspective the chief practical problem is recruiting reviewers

they don’t want to alienate the reviewers by telling them that half their advice sucks in their cover letter

Responding to a research question by brainstorming possibly relevant citations or methods

When I give you advice as a peer reviewer there is a strong presumption that you take the advice even if it’s mediocre

When I give you advice in my office you can ask follow-up questions

When I give advice as a peer reviewer it’s up to you to hope that you read the entrails in a way that correctly augurs the will of the peer reviewers.

Being specific has the ancillary benefit that it’s costly to the reviewer which should help you maintain the discipline to thin the mindfart herd stampeding into the authors’ revisions.

When I give you advice about your new project you can use it to shape the way the project develops organically. When I give it to you as a reviewer you can only graft it on after the fact.

time is money. It usually takes me an amount of time that is at least the equivalent of a course release to turn-around an R&R and at most schools a course release in turn is worth about $10,000 to $30,000 if you’re lucky enough to raise the grants to buy them.

A lot of review comments ultimately boil down to some variation on “this reminds me of this citation” or “this research agenda could go in this direction.” OK, great. Now ask yourself, is it a problem that this paper does not yet do these things or are these just possibilities you want to share with the author?

As a related issue, demonstrate some rhetorical humility.

On quite a few methodological and theoretical issues there is a reasonable range of opinion. Don’t force the author to weigh in on your side.

One reaction to this would be to say the McPherson et al paper is refuted and ought not be cited. However Petev summarizes the controversy in footnote 10 and then in footnote 17 explains why his own data is a semi-independent (same dataset, different variables) corroboration of McPherson et al.

Theoretical debates are rarely an issue of decisive refutation or strictly cumulative knowledge but rather at any given time there’s a reasonable range of opinions and you shouldn’t demand that the author go with your view but at most that they explore its implications if they were to.

There are cases where you fall on one side of a theoretical or methodological gulf and the author on another to the extent that you feel that you can’t really be fair.

you as the reviewer have to decide if you’re going to engage in what philosophers of science call “the demarcation problem” and sociologists of science call “boundary work” or you’re going to recuse yourself from the review.

The theory section is not about demonstrating basic competence or reciting a creedal confession and so it does not need to discuss every book or article ever published on the subject or even just the things important enough to appear on your graduate syllabus or field exam reading list.

If the submission reminds you of a citation that’s relevant to the author’s subject matter, think about whether it would materially affect the argument.

You could get the same information with a week's sorting of SEC filings. But instead, you have just circumvented that laborious process by going straight to just one of the "meta-narratives" that form the superstructure of journalism.

The currency of language really is the collection of what might be called the "meta-stories."

There's nothing wrong with this. For the art of storytelling -- journalism, that is -- is essentially unchanged from the tale-telling of Neolithic shamans millennia ago up through and including today's New York Times. Cultural anthropologists will explain that our brains are wired for this. So be it.

We at Stratfor may not "sync up." Journalists certainly do.

"Meta-data," as the technologists call it, is more simply understood as "data about data."

Where the online battle for eyeballs becomes truly epic, however, (Google "the definition of epic" for yet another storyteller's meta-story) is when these series of tags are organized into a form of meta-data called a "taxonomy."

And thus we arrive at the outermost layer of the media's skin in our emerging and interconnected age. This invisible skin over it all comes in the form of a new term of art, "search engine optimization," or in the trade just "SEO."

With journalists already predisposed by centuries of convention to converge on stories knitted from a common canon, the marriage of meta-narrative and meta-data simply accelerates to the speed of light the calibration of topic and theme.

For by the doctrines of the Internet's new commercial religion, a move disrupting the click stream was -- and is -- pure heresy. But our readers still need to know about Colombia, just as they need our unique perspectives on Syria.

But, perhaps the most devastating impact on traditional punditry: politics and campaigning has a relatively small impact on elections. According to Silver’s model, Obama had a strong likelihood of winning several months before the election. Elections favor incumbents and Romney was an uncharismatic opponent, who wasn’t all that well liked even within his own party. Other influential factors, such as the economy, are completely outside the control of campaigns. The economy picked up before the election. Any conservative challenger had an uphill battle.

Maybe sometimes it’s the questions that are biased, not the answers,” he said, flashing a friendly smile.

That question has been central to Ioannidis’s career. He’s what’s known as a meta-researcher, and he’s become one of the world’s foremost experts on the credibility of medical research.

He charges that as much as 90 percent of the published medical information that doctors rely on is flawed.

“I take all the researchers who visit me here, and almost every single one of them asks the tree the same question,” Ioannidis tells me, as we contemplate the tree the day after the team’s meeting. “‘Will my research grant be approved?’” He chuckles, but Ioannidis (pronounced yo-NEE-dees) tends to laugh not so much in mirth as to soften the sting of his attack. And sure enough, he goes on to suggest that an obsession with winning funding has gone a long way toward weakening the reliability of medical research.

“I assumed that everything we physicians did was basically right, but now I was going to help verify it,” he says. “All we’d have to do was systematically review the evidence, trust what it told us, and then everything would be perfect.” It didn’t turn out that way. In poring over medical journals, he was struck by how many findings of all types were refuted by later findings. Of course, medical-science “never minds” are hardly secret. And they sometimes make headlines, as when in recent years large studies or growing consensuses of researchers concluded that mammograms, colonoscopies, and PSA tests are far less useful cancer-detection tools than we had been told; or when widely prescribed antidepressants such as Prozac, Zoloft, and Paxil were revealed to be no more effective than a placebo for most cases of depression; or when we learned that staying out of the sun entirely can actually increase cancer risks; or when we were told that the advice to drink lots of water during intense exercise was potentially fatal; or when, last April, we were informed that taking fish oil, exercising, and doing puzzles doesn’t really help fend off Alzheimer’s disease, as long claimed. Peer-reviewed studies have come to opposite conclusions on whether using cell phones can cause brain cancer, whether sleeping more than eight hours a night is healthful or dangerous, whether taking aspirin every day is more likely to save your life or cut it short, and whether routine angioplasty works better than pills to unclog heart arteries.

“I realized even our gold-standard research had a lot of problems,” he says.

This array suggested a bigger, underlying dysfunction, and Ioannidis thought he knew what it was. “The studies were biased,” he says. “Sometimes they were overtly biased. Sometimes it was difficult to see the bias, but it was there.” Researchers headed into their studies wanting certain results—and, lo and behold, they were getting them. We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously. “At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,” says Ioannidis. “There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded.”

He chose to publish one paper, fittingly, in the online journal PLoS Medicine, which is committed to running any methodologically sound article without regard to how “interesting” the results may be. In the paper, Ioannidis laid out a detailed mathematical proof that, assuming modest levels of researcher bias, typically imperfect research techniques, and the well-known tendency to focus on exciting rather than highly plausible theories, researchers will come up with wrong findings most of the time.

The article spelled out his belief that researchers were frequently manipulating data analyses, chasing career-advancing findings rather than good science, and even using the peer-review process—in which journals ask researchers to help decide which studies to publish—to suppress opposing views.

Ioannidis was putting his contentions to the test not against run-of-the-mill research, or even merely well-accepted research, but against the absolute tip of the research pyramid. Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been retested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable. That article was published in the Journal of the American Medical Association.

When a five-year study of 10,000 people finds that those who take more vitamin X are less likely to get cancer Y, you’d think you have pretty good reason to take more vitamin X, and physicians routinely pass these recommendations on to patients. But these studies often sharply conflict with one another. Studies have gone back and forth on the cancer-preventing powers of vitamins A, D, and E; on the heart-health benefits of eating fat and carbs; and even on the question of whether being overweight is more likely to extend or shorten your life. How should we choose among these dueling, high-profile nutritional findings? Ioannidis suggests a simple approach: ignore them all.

the odds are that in any large database of many nutritional and health factors, there will be a few apparent connections that are in fact merely flukes, not real health effects—it’s a bit like combing through long, random strings of letters and claiming there’s an important message in any words that happen to turn up.

nd these problems are aside from ubiquitous measurement errors (for example, people habitually misreport their diets in studies), routine misanalysis (researchers rely on complex software capable of juggling results in ways they don’t always understand), and the less common, but serious, problem of outright fraud (which has been revealed, in confidential surveys, to be much more widespread than scientists like to acknowledge).

And so it goes for all medical studies, he says. Indeed, nutritional studies aren’t the worst. Drug studies have the added corruptive force of financial conflict of interest. The exciting links between genes and various diseases and traits that are relentlessly hyped in the press for heralding miraculous around-the-corner treatments for everything from colon cancer to schizophrenia have in the past proved so vulnerable to error and distortion, Ioannidis has found, that in some cases you’d have done about as well by throwing darts at a chart of the genome.

Though scientists and science journalists are constantly talking up the value of the peer-review process, researchers admit among themselves that biased, erroneous, and even blatantly fraudulent studies easily slip through it.

Of those 45 super-cited studies that Ioannidis focused on, 11 had never been retested. Perhaps worse, Ioannidis found that even when a research error is outed, it typically persists for years or even decades. He looked at three prominent health studies from the 1980s and 1990s that were each later soundly refuted, and discovered that researchers continued to cite the original results as correct more often than as flawed—in one case for at least 12 years after the results were discredited.

Medical research is not especially plagued with wrongness. Other meta-research experts have confirmed that similar issues distort research in all fields of science, from physics to economics (where the highly regarded economists J. Bradford DeLong and Kevin Lang once showed how a remarkably consistent paucity of strong evidence in published economics studies made it unlikely that any of them were right).

Ioannidis initially thought the community might come out fighting. Instead, it seemed relieved, as if it had been guiltily waiting for someone to blow the whistle, and eager to hear more. David Gorski, a surgeon and researcher at Detroit’s Barbara Ann Karmanos Cancer Institute, noted in his prominent medical blog that when he presented Ioannidis’s paper on highly cited research at a professional meeting, “not a single one of my surgical colleagues was the least bit surprised or disturbed by its findings.” Ioannidis offers a theory for the relatively calm reception. “I think that people didn’t feel I was only trying to provoke them, because I showed that it was a community problem, instead of pointing fingers at individual examples of bad research,” he says. In a sense, he gave scientists an opportunity to cluck about the wrongness without having to acknowledge that they themselves succumb to it—it was something everyone else did.

The irony of his having achieved this sort of success by accusing the medical-research community of chasing after success is not lost on him, and he notes that it ought to raise the question of whether he himself might be pumping up his findings.

Tatsioni doesn’t so much fear that someone will carve out the man’s healthy appendix. Rather, she’s concerned that, like many patients, he’ll end up with prescriptions for multiple drugs that will do little to help him, and may well harm him. “Usually what happens is that the doctor will ask for a suite of biochemical tests—liver fat, pancreas function, and so on,” she tells me. “The tests could turn up something, but they’re probably irrelevant. Just having a good talk with the patient and getting a close history is much more likely to tell me what’s wrong.” Of course, the doctors have all been trained to order these tests, she notes, and doing so is a lot quicker than a long bedside chat. They’re also trained to ply the patient with whatever drugs might help whack any errant test numbers back into line.

patients often don’t even like it when they’re taken off their drugs, she explains; they find their prescriptions reassuring.

Already feeling that they’re fighting to keep patients from turning to alternative medical treatments such as homeopathy, or misdiagnosing themselves on the Internet, or simply neglecting medical treatment altogether, many researchers and physicians aren’t eager to provide even more reason to be skeptical of what doctors do—not to mention how public disenchantment with medicine could affect research funding.

being wrong in science is fine, and even necessary

Mr. Rushdie started using a computer only when the Ayatollah Khomeini’s 1989 fatwa drove him underground. “My writing has got tighter and more concise because I no longer have to perform the mechanical act of re-typing endlessly,” he explained during an interview while in hiding. “And all the time that was taken up by that mechanical act is freed to think.”

At the Emory exhibition, visitors can log onto a computer and see the screen that Mr. Rushdie saw, search his file folders as he did, and find out what applications he used. (Mac Stickies were a favorite.) They can call up an early draft of Mr. Rushdie’s 1999 novel, “The Ground Beneath Her Feet,” and edit a sentence or post an editorial comment.

To the Emory team, simulating the author’s electronic universe is equivalent to making a reproduction of the desk, chair, fountain pen and paper that, say, Charles Dickens used, and then allowing visitors to sit and scribble notes on a copy of an early version of “Bleak House.”

The heart of the lab is the Forensic Recovery of Evidence Device, nicknamed FRED, which enables archivists to dig out data, bit by bit, from current and antiquated floppies, CDs, DVDs, hard drives, computer tapes and flash memories, while protecting the files from corruption.

The third category is the nitty-gritty, “Core Ideas in Four Disciplinary Areas,” namely “Physical Sciences,” “Life Sciences,” “Earth and Space Sciences,” and “Engineering, Technology, and the Applications of Science.”

Don Quixote is a malevolent universe attack on all values as such. It belongs in the same class with two other books, which together make up the three books I hate most: Don Quixote, Anna Karenina, and Madame Bovary.

In going through Rand's aesthetic judgments, one can't help noticing how often Rand conflates her personal tastes with objective truth

The standard of truth and morality in Objectivism is not "reason" or logic or fact; it is Ayn Rand herself. What Rand said is true is true, despite what all the great thinkers and scientists said before her. What Ayn Rand said is good or evil is good or evil, regardless of whatever natural needs may exist elsewhere in the universe. This explains, perhaps more than anything else, why Objectivsm so quickly degenerated into an Ayn Rand personality cult.

If Derbyshire were a lone crank, none of it would matter much. But he’s not.

they see them selves as advocates of a sort of enlightened racism that doesn’t shrink from calling a spade a spade but isn’t inherently unjust.

They could care less about skin color, they say; it really is the content of people’s characters that concerns them, and that content really does suffer more in blacks than whites.

First, a history of slavery, segregation, and (yes) racism, means that African American communities suffer from some social problems at higher rates than whites.

The argument that Derbyshire and those like him make is that we are justified in treating an entire population as a threat – in essentially shunning them in the most degrading way – because one’s chances of being harmed by any given member of that population, while very low, is not quite as low as one’s chances of being harmed by the general population.

Second, there is the issue of character: because this, after all, is what really motivates these attempts at establishing an enlightened racism that gives individuals the benefit of the doubt while acknowledging the truth about general cultural differences.

If they respond with sociological data about education and birth rates and all the rest, we only have to respond that like crime rates, they’re exactly the sort of consequences one would expect from a history of oppression and even then fail to justify racist stereotypes.

And so they need to disabuse themselves of the idea that true, unenlightened racism is a term very narrowly defined: that it requires a personal hatred of individual black people based on their skin color despite evidence of redeeming personal qualities.

What they think of as redeeming personal qualities are just qualities that tend to make them less uncomfortable. But the hatred of black culture and post-hoc rationalizations of this hatred using sociological data are just what racism is.

This is not to say that mere discomfort with cultural difference is the same thing as racism (or xenophobia). Such discomfort is unavoidable: You’d have this sort of discomfort if you tried live in a foreign country for a while, and you’d be tempted by the same sorts of ideas about how stupid and mean people are for not doing things the way you’re used to.

That lack of reflection is felt not only as a distressing deprivation of social oxygen, but as an affront, a positive discourtesy.