Science and gun violence: why is the research so weak? [Part 2] - Boing Boing - 1 views
boingboing.net/...guns.html
social science results methods experiment test research modeling assumption global warming guns public policy
![](/images/link.gif)
-
Scientists are missing some important bits of data that would help them better understand the effects of gun policy and the causes of gun-related violence. But that’s not the only reason why we don’t have solid answers. Once you have the data, you still have to figure out what it means. This is where the research gets complicated, because the problem isn’t simply about what we do and don’t know right now. The problem, say some scientists, is that we —from the public, to politicians, to even scientists themselves—may be trying to force research to give a type of answer that we can’t reasonably expect it to offer. To understand what science can do for the gun debates, we might have to rethink what “evidence-based policy” means to us.
-
For the most part, there aren’t a lot of differences in the data that these studies are using. So how can they reach such drastically different conclusions? The issue is in the kind of data that exists, and what you have to do to understand it, says Charles Manski, professor of economics at Northwestern University. Manski studies the ways that other scientists do research and how that research translates into public policy.
-
Even if we did have those gaps filled in, Manski said, what we’d have would still just be observational data, not experimental data. “We don’t have randomized, controlled experiments, here,” he said. “The only way you could do that, you’d have to assign a gun to some people randomly at birth and follow them throughout their lives. Obviously, that’s not something that’s going to work.”
- ...14 more annotations...
-
This means that, even under the best circumstances, scientists can’t directly test what the results of a given gun policy are. The best you can do is to compare what was happening in a state before and after a policy was enacted, or to compare two different states, one that has the policy and one that doesn’t. And that’s a pretty inexact way of working.
-
Add in enough assumptions, and you can eventually come up with an estimate. But is the estimate correct? Is it even close to reality? That’s a hard question to answer, because the assumptions you made—the correlations you drew between cause and effect, what you know and what you assume to be true because of that—might be totally wrong.
-
It’s hard to tease apart the effect of one specific change, compared to the effects of other things that could be happening at the same time.
-
This process of taking the observational data we do have and then running it through a filter of assumptions plays out in the real world in the form of statistical modeling. When the NAS report says that nobody yet knows whether more guns lead to more crime, or less crime, what they mean is that the models and the assumptions built into those models are all still proving to be pretty weak.
-
From either side of the debate, he said, scientists continue to produce wildly different conclusions using the same data. On either side, small shifts in the assumptions lead the models to produce different results. Both factions continue to choose sets of assumptions that aren’t terribly logical. It’s as if you decided that anybody with blue shoes probably had a belly-button piercing. There’s not really a good reason for making that correlation. And if you change the assumption—actually, belly-button piercings are more common in people who wear green shoes—you end up with completely different results.
-
The Intergovernmental Panel on Climate Change (IPCC) produces these big reports periodically, which analyze lots of individual papers. In essence, they’re looking at lots of trees and trying to paint you a picture of the forest. IPCC reports are available for free online, you can go and read them yourself. When you do, you’ll notice something interesting about the way that the reports present results. The IPCC never says, “Because we burned fossil fuels and emitted carbon dioxide into the atmosphere then the Earth will warm by x degrees.” Instead, those reports present a range of possible outcomes … for everything. Depending on the different models used, different scenarios presented, and the different assumptions made, the temperature of the Earth might increase by anywhere between 1.5 and 4.5 degrees Celsius.
-
What you’re left with is an environment where it’s really easy to prove that your colleague’s results are probably wrong, and it’s easy for him to prove that yours are probably wrong. But it’s not easy for either of you to make a compelling case for why you’re right.
-
Statistical modeling isn’t unique to gun research. It just happens to be particularly messy in this field. Scientists who study other topics have done a better job of using stronger assumptions and of building models that can’t be upended by changing one small, seemingly randomly chosen detail. It’s not that, in these other fields, there’s only one model being used, or even that all the different models produce the exact same results. But the models are stronger and, more importantly, the scientists do a better job of presenting the differences between models and drawing meaning from them.
-
“Climate change is one of the rare scientific literatures that has actually faced up to this,” Charles Manski said. What he means is that, when scientists model climate change, they don’t expect to produce exact, to-the-decimal-point answers.
-
“It’s been a complete waste of time, because we can’t validate one model versus another,” Pepper said. Most likely, he thinks that all of them are wrong. For instance, all the models he’s seen assume that a law will affect every state in the same way, and every person within that state in the same way. “But if you think about it, that’s just nonsensical,” he said.
-
On the one hand, that leaves politicians in a bit of a lurch. The response you might mount to counteract a 1.5 degree increase in global average temperature is pretty different from the response you’d have to 4.5 degrees. On the other hand, the range does tell us something valuable: the temperature is increasing.
-
The problem with this is that it flies in the face of what most of us expect science to do for public policy. Politics is inherently biased, right? The solutions that people come up with are driven by their ideologies. Science is supposed to cut that Gordian Knot. It’s supposed to lay the evidence down on the table and impartially determine who is right and who is wrong.
-
Manski and Pepper say that this is where we need to rethink what we expect science to do. Science, they say, isn’t here to stop all political debate in its tracks. In a situation like this, it simply can’t provide a detailed enough answer to do that—not unless you’re comfortable with detailed answers that are easily called into question and disproven by somebody else with a detailed answer.
-
Instead, science can reliably produce a range of possible outcomes, but it’s still up to the politicians (and, by extension, up to us) to hash out compromises between wildly differing values on controversial subjects. When it comes to complex social issues like gun ownership and gun violence, science doesn’t mean you get to blow off your political opponents and stake a claim on truth. Chances are, the closest we can get to the truth is a range that encompasses the beliefs of many different groups.