Thanks, Mitt Romney! Photographer: Neal Hamberg/Bloomberg
Thanks, Mitt Romney! Photographer: Neal Hamberg/Bloomberg

Last May, researchers studying a randomized controlled trial of Oregon’s Medicaid expansion released a bombshell report that found no significant improvements in mortality rates or key health markers for blood pressure, hypertension and diabetes control. I did three major posts on it (one of which was a guest post from Jim Manzi, the world’s leading popular author on the virtues -- and limitations -- of RCTs). At the end of my first post, I wrote:

At this point, the only two large-scale randomized control trials that we have done on the benefits of paying for peoples' health care have both come back showing surprisingly small effects. In 2011, when the first results came out of Oregon, that was not what Obamacare's supporters were predicting. They were predicting that the second phase of the Oregon study would show large, significant effects on basic health measures like blood pressure, blood sugar, and cholesterol control. It didn't. There's really no other way to put it.

A good Bayesian -- and aren't most of us are supposed to be good Bayesians these days? -- should be updating in light of this new information. Given this result, what is the likelihood that Obamacare will have a positive impact on the average health of Americans? Every one of us, for or against, should be revising that probability downwards. I'm not saying that you have to revise it to zero; I certainly haven't. But however high it was yesterday, it should be somewhat lower today.

Well, good Bayesians have another data point: A different team of researchers has looked at mortality rates in Massachusetts after Romneycare. And they found what is, for mortality rates, a huge decrease: 8.2 per 100,000 per year for adults 20 to 64. Doesn’t sound like a lot, but your risk of dying in any year between 20 and 64 isn’t that high. So a big, fat decline like 8.2 per 100,000 makes you sit up and take notice.

I’m traveling this week, so you’re unlikely to get three posts on the subject. But here’s my basic take: The study looks pretty good. It is hard to reconcile these results with Oregon. Good Bayesians will update their beliefs accordingly. People who are just looking for studies that confirm what they believe will either celebrate this study and find reasons to discount the Oregon results, or vice versa.

I can think of reasons to believe Oregon rather than Massachusetts: It’s a real RCT, rather than a comparison of counties that are supposed to be similar but might not actually have been in practice. There are surprising results that might be red flags about the data: For instance, cancer mortality dropped, even though the study only ran four years. If the benefit is coming from early detection, I would expect it to take longer to have a significant impact on the data. And if it is coming from chemotherapy and radiation treatments, then cutting-edge chemotherapy and radiation treatments are much more effective than studies had previously led me to believe. Corollary: Hog-wild American-style spending on expensive cancer treatments is a lifesaver, and we will and should probably ignore any calls for European-style cost control.

Massachusetts also has differently sized samples -- the control group has about 10 times the population of the treatment group. Since larger groups should tend to have less variance, this might increase the odds of getting a surprisingly good or surprisingly bad result with your treatment group.

Most worrying is the period they compared: 2007 to 2010. They started with a sample of counties matched for employment and demographics. But look what happened to Massachusetts versus the rest of the country during the financial crisis:

The authors are not stupid; they control for unemployment and poverty. But if the financial crisis affected Massachusetts differently from the rest of the country -- and it did -- then that may well show up in mortality in ways that are hard to measure. Some support for this thesis comes from this figure, which shows mortality in Massachusetts dropping especially far in 2008, while rising slightly in the control group:

Source: Annals of Internal Medicine
Source: Annals of Internal Medicine

That said, I can also think of reasons to believe Massachusetts over Oregon. The treatment group sample is smaller than the control group by quite a lot, but it’s still huge by comparison with Oregon: 3,900,000 rather than around 6,000. And Massachusetts is a single comprehensive system, unlike Medicaid, which people might drop in and out of as their income changes, dampening the effect of insurance. The authors find mortality that’s amenable to treatment dropping more rapidly than overall mortality. And comparisons with nearby counties, and with the state’s elderly population, also deliver the same result.

So what should we believe? I haven’t changed my beliefs radically: I still basically think that health insurance improves mortality rates, but that that improvement is unlikely to be huge if you can get results like Oregon. However, after yesterday’s report, I’ve revised the probability of “huge benefits” upward, and you should do the same. And beware of those who are only willing to revise their beliefs in one direction.

To contact the author of this article: Megan McArdle at mmcardle3@bloomberg.net.

To contact the editor responsible for this article: James Gibney at jgibney5@bloomberg.net.