Is there hope for evidence-based policy?
On research, failure, and progress: A special forum with Vital City
The third issue of Hypertext, a collaboration with our friends at Vital City magazine, features a star-studded cast of authors contemplating the prospects for “evidence-based policymaking” over 19 (count ‘em) essays.
The notion that policy decisions can be guided by sober, scientific analysis of “what works” is widely attractive, and indeed, it’s an important component of the Niskanen Center’s advocacy.
But what if the best evidence suggests that virtually nothing policymakers are trying moves the needle?
That is the central claim of a new and hotly debated article on criminal justice policy by Megan Stevenson, an economist at the University of Virginia’s law school: “Cause, effect, and the structure of the social world.” The article, which is well worth reading in its entirety, surveys the literature on randomized controlled trials (RCTs) of criminal-justice interventions. Stevenson argues that the vast majority of interventions evaluated with this “gold-standard” method “have little to no lasting effect” and this tells us something important about the reach of policy. “When it comes to the type of limited-scope interventions that lend themselves to high-quality evaluation,” she writes, “social change is hard to engineer.”
Is it really that bad? Is the whole evidence-based model misguided? Must we act more boldly — and can we? Are policymakers back to navigating by ideology and gut?
We and our partners at Vital City asked leading lights from academia, philanthropy, advocacy, government, and journalism to weigh in — and they gave us a lot to chew on. To highlight just a few of our terrific contributions:
What can we know and how can we know it? John MacDonald and Kerri Raissian warn against simply reading the RCT literature as a counsel of despair. Meanwhile, John Maki argues that public policy has an irreducibly interpersonal dimension that is almost impossible to pick up on with quantitative studies. Jonathan Rauch agrees: “RCTs have their place — and we should keep them in it.”
Incrementalism and failure are the way of the world. Aaron Chalfin argues that RCTs accurately reflect the difficulty of changing the world, but this hardly means progress is impossible. Matt Grossmann points out that progressives and conservatives interpret the failures of “evidence-based policy” in opposing directions, and argues incrementalism is a product of political constraints that cannot be wished away. And the alternative, Philip J. Cook and Jens Ludwig add, is recklessness: “You only live once, but for how long?”
Meanwhile, research and policy do get better. John Arnold notes that the bar on research quality is rising, and useful findings are emerging as a result. Jen Doleac explains the key improvements we have made in recent years and calls for a “science of scaling” for successful interventions. Jeffrey Liebman takes a more pessimistic view of the literature but calls for more experimentation, not less, and counsels against mandating “evidence-based policy” in ways that lock in programs with weak support.
There are concrete takeaways for policymakers. Alex Tabarrok argues that changing people’s preferences rarely works - but changing their incentives and constraints can. Jen Pahlka notes that if policy change requires ongoing, laborious cultivation to succeed, the same is true of the bureaucracies pursuing that change. Richard Hahn envisions a bureaucracy that wrestles away the academic monopoly on rigorous evaluation to learn faster.
Finally, Tracy Palandjian and Jake Segal offer some valuable perspective: “We see good evaluations … as momentary glimpses of underlying truths about the world, the tips of icebergs poking out over a dark sea. They shift under our feet the moment we step — nothing is ever ‘proven’ — but it’s better than jumping, unseeing, into the water.”
Please visit Hypertext today to start browsing the entire issue, and look for individual essays in your inbox every few days over the coming weeks.
As always, please spread the word about Hypertext and help us broaden these critical conversations. And keep in mind that we publish response essays — if you know someone who wants to keep the debate going, have them reach out. Finally, don’t forget to sign up for updates from our wonderful partners at Vital City, who are building a hub for thinkers and doers to keep making policy better.