Group items tagged

Statistical validation of results, as Shaffer described it, simply involves testing the null hypothesis: that the pattern you detect in your data occurs at random. If you can reject the null hypothesis—and science and medicine have settled on rejecting it when there's only a five percent or less chance that it occurred at random—then you accept that your actual finding is significant. The problem now is that we're rapidly expanding our ability to do tests. Various speakers pointed to data sources as diverse as gene expression chips and the Sloan Digital Sky Survey, which provide tens of thousands of individual data points to analyze. At the same time, the growth of computing power has meant that we can ask many questions of these large data sets at once, and each one of these tests increases the prospects than an error will occur in a study; as Shaffer put it, "every decision increases your error prospects." She pointed out that dividing data into subgroups, which can often identify susceptible subpopulations, is also a decision, and increases the chances of a spurious error. Smaller populations are also more prone to random associations. In the end, Young noted, by the time you reach 61 tests, there's a 95 percent chance that you'll get a significant result at random. And, let's face it—researchers want to see a significant result, so there's a strong, unintentional bias towards trying different tests until something pops out. Young went on to describe a study, published in JAMA, that was a multiple testing train wreck: exposures to 275 chemicals were considered, 32 health outcomes were tracked, and 10 demographic variables were used as controls. That was about 8,800 different tests, and as many as 9 million ways of looking at the data once the demographics were considered.

The North American Spine Society and the Society of Interventional Radiology have pointed to flaws in both studies. And earlier studies, published over 15 years, found major benefits to kyphoplasty and a related procedure called vertebroplasty. "We're missing opportunities for patients to receive a safe and effective treatment that can significantly reduce their pain and disability," said Malamis, an interventional radiologist. The procedures are used to treat vertebral compression fractures in patients with osteoporosis and other conditions that result in brittle bones. In a vertebroplasty, an acrylic cement is injected into a fractured vertebra. In a kyphoplasty, a balloon-tipped catheter first is inserted into the fracture. The balloon is inflated to restore the height and shape of the vertebra before the cement is injected. Neva Nelson, 74, of Naperville, Ill., said a kyphoplasty that Malamis performed in October, 2009, has greatly reduced her pain in a vertebra in her lower back that she fractured after falling on ice. Before her kyphoplasty, Nelson had to sit on cushions. Walking, and especially standing, were painful. "I had to do something," she said. "I could not go on like that." Nelson said that since undergoing her kyphoplasty, "I don't have to worry about my back any more." In the controversial studies, patients were randomly assigned to receive a vertebroplasty or a placebo-like "sham" procedure. In the sham procedure, patients received an injection of anesthetic, but no cement. However, patients in severe pain are reluctant to enroll in a trial where there's a 50 percent chance of receiving a sham treatment. In one of the studies, researchers had to screen 1,813 patients to enroll just 131 subjects. In the other study, only 78 of 219 eligible patients were enrolled. This low enrollment rate raises the possibility that the patients who did enroll were not representative. Patients experience the greatest pain during the first three months after a compression fracture. Thereafter, pain gradually subsides. Thus, a vertebroplasty or kyphoplasty provides the greatest benefit when performed within a week or two of the fracture. But the studies enrolled patients up to 12 months after fractures. In addition to reducing pain and disability, a kyphoplasty can reduce the risk of subsequent fractures by improving the angle and height of the spine. The studies evaluated vertebroplasty alone, and did not include the more innovative and very different kyphoplasty procedure. Malamis suggests the medical community wait for the results of additional studies now underway before passing final judgment on vertebroplasty or kyphoplasty. In the mean time, he notes that Medicare still covers the procedures.