The Barack Obama administration has finally told the news media what metrics it will be using to evaluate HealthCare.gov's performance. Juliet Eilperin and Amy Goldstein of the Washington Post report that it is looking for 80 percent of users to be able to buy health-care plans online:
To assess progress toward the goal, administration officials have developed two new measurements, appearing in reports generated each morning, that show how long consumers must wait for pages to load on HealthCare.gov and how often they get error messages, government and industry officials said.
The “vast majority” phrase has been invoked repeatedly by Obama, [Jeffrey] Zients and other administration officials, with little explanation of what it means. In a news conference Thursday, while apologizing for the health-care law’s rollout problems, Obama said that the target was to ensure that by the end of the month that “the majority of people who use it will be able to see it operate the way it was supposed to.” ...
According to a government official familiar with the new target, the 20 percent who are unlikely to be able to enroll online are expected to fall into three groups: people whose family circumstances are so complicated that the Web site cannot determine their eligibility for subsidies to help pay for health plans; people uncomfortable buying insurance on a computer; and people who encounter technical problems on the Web site.
This is still a lot vaguer than I’d like. For starters, are we saying that the 80 percent will be able to enroll in one shot? Or is this “80 percent by March 31”? In other words, are we measuring 80 percent of attempts to use the website or 80 percent of the folks who will ultimately enroll on the exchange?
The article says that the administration has related metrics for things such as lag time and the error rates in delivering pages, which imply that the answer is “80 percent of attempts will succeed.” But then why would you lump “people who are uncomfortable buying insurance on a computer” into the group of people who won’t succeed? Presumably, something close to none of the attempts to use the website to buy insurance will be made by people who are uncomfortable buying insurance on a website.
"Eighty percent of total enrollees having gone through the site" is a pretty useless metric; the administration could theoretically satisfy this metric by getting five people enrolled: four on the exchanges and one through a paper application.
"Eighty percent of attempts" would tell us more about the functionality of the exchange. But it’s almost too good a metric -- I doubt that 80 percent of attempts would be completed even if the website were working perfectly. A lot of people are going to get partway through the process and quit -- many of them intending to come back later, some because they’ve discovered that they can’t get as good a deal on the exchange as they can through Cobra or their employer. So it’s unlikely that the administration is measuring this directly; rather, they’d be constructing a model based on lower-level metrics such as ... lag times and page-delivery errors. The nice thing about models is, of course, that they can be tweaked. All of which is to say that if the administration does not want to fail this metric, it won’t.
And neither metric tells us about the accuracy of the data going to insurers. Needless to say, if that part isn’t working, it won’t be much comfort that “the majority of people who use it ... see it operate the way it was supposed to,” because they can’t see the system mangling their enrollment information and leaving them uninsured.
All the vagueness probably helps the administration. When your project is having difficulties, the ideal performance measurement is a relatively easy metric that sounds like something much more stringent. I once did work for a client whose previous consultant had insisted that his promises of “less than 1 percent server down time” had been accurate because the server was on, even if it was refusing to let any users access its files. Amazingly, the client went along with this for almost two months before firing him. In its defense, the company was run by English majors.*
Ultimately, there’s only one metric that really matters: enrollment. Are a lot of people signing up? And does that number contain a good percentage of young, healthy people, or is it mostly the old and the sick? Is their information being correctly transmitted to insurers?
We’ll have a better idea about this when the next round of enrollment figures is released in December -- well, unless the administration also chooses weirdly vague metrics for enrollment.
But even if we get hard numbers, early December is leaving it awfully late; folks who are losing their current policies need to buy new ones by Dec. 15. If the system is really and truly broken, there's a real risk that we’ll only know that for certain long past the point where it’s too late to do anything about it.
*Put down the poison pen; I was an English major.