The Real Cost of Fake Statistics

Jerry Z. Muller, The Tyranny of Metrics

We know the horror stories: schoolteachers ignoring larger goals and teaching for the test. Universities needing to create new administrative positions to handle the paperwork, then offsetting the cost by packing departments with adjunct instructors. Financiers booking profits for this quarter, while punting debits to the next quarter. Police failing to report serious crimes. Tales of trusted professionals manipulating statistics have become downright legendary.

Economic historian Jerry Z. Muller doesn’t unilaterally oppose metrics and other attempts to quantify abstract goals. They have important value, he writes, when applied appropriately. When used for internal analysis, say, or in low-stakes testing to determine progress on clearly defined goals, statistics can have powerful beneficial effects. People love metrics because, at first, they provide important information that improves our ability to pursue our purposes.

Problems arise, Muller writes, because people who aren’t familiar with important fields, believe they can stretch short-term benefits of metrics infinitely into the future. When small, underfunded schools’ funding, even their existence, gets threatened because students cannot ace standardized tests written in another state, for instance. When politicians intimidate police into keeping crime stats low. When hospitals falsify medical records to avoid Medicaid decertification.

Modern technocratic societies began demanding elaborate metrics to measure the immeasurable, according to Muller, for completely understandable reasons. New research in psychology and behavioral economics has demonstrated the subjectivity of human judgement. The demand for “transparency” in public affairs means people want abstruse topics translated into layperson language. And little seems more objective and transparent than plain, old-fashioned numbers.

In practice, though, this isn’t true. First, the metrics most often measure the areas easiest to measure, not what matters most. Schools measure questions with obviously right answers, not students’ ability to reason on nuanced or debatable topics. Financiers receive bonuses for sheer money made, regardless of stability. Militaries count bodies killed, not hearts and minds won. This results in very short-term thinking and reaching for low-hanging fruit.

This distorts the purposes served by “transparency,” and its close cousin, “accountability.” (Muller makes a persuasive case that transparency frequently isn’t desirable anyway, since it’s the opposite of nuance.) Knowing their funding and careers hang precariously, officials tasked with maintaining the metrics start gaming the system, chasing desirable outcomes. Safe streets, healed patients, and an educated citizenry become secondary to cramming the numbers.

Jerry Z. Muller

Further, this demand for metrics creates amassive mandatory bureaucracy. Government demands for more numbers from schools, universities, and hospitals have forced them to hire actuaries whose sole job is to write and refine government statistics. It’s become axiomatic recently to complain about the increasing number of administrators in public institutions. But they need those administrators to fulfill their accountability mandate—and the government needs bureaucrats to read the resulting numbers.

This comes at the expense of seniority and experience. Muller writes, correctly, that serious science calls human judgement abilities into question. But metrics are scarcely better, since they apply the judgement of the metrics writers—who frequently are outsiders to the field they measure—over the judgement of those who dedicate their lives to work, study, or practice in a field. For all its flaws, human discretion is invaluable when it originates from time-tested skills.

And don’t cite “run government like a business” here. Muller demonstrates, extensively, that corporations are just as procedural, and just as prone to distort numbers for indefensible reasons.

I’d go beyond Muller. When enormous bureaucratic institutions demand conformity to numbers, those numbers generally reflect the beige middle ground. Thus public schools with substantially Black or Hispanic student bodies, American aid agencies working in non-Western countries, and doctors treating patients ill-equipped to monitor their own wellness, eventually force the poor, the non-white, and the international, to conform to middle-class White American standards that don’t apply.

Muller isn’t entirely pessimistic about metrics. From the beginning, he studiously distinguishes misapplications of statistics, the kind of horror stories we see on the nightly news, from correctly applied numbers. He praises appropriately applied numbers, particularly those used to measure internally, as read by experienced minds. If metrics measure what matters, and people accept metrics’ limits, numbers can have powerful beneficial effects.

Muller extensively cites Campbell’s Law, which, paraphrased, states that statistics, when misapplied, always distort what they measure. Though recent studies have demonstrated this apparent truism, we keep forgetting it. Clear back in the Victorian era, the poet and educator Matthew Arnold decried how top-down measurement efforts damage what they pretend to support. Maybe, by easing up on misapplied statistics, we can improve outcomes for everybody.