The limits of Death Star thinking
Why we need a new theory of change.
The observation that most “evidence-based interventions” deliver disappointing results rhymes with something I’ve been observing in my work to improve government capacity — namely, that our entire system of law, policy and regulation is assumed to be mechanistic. Do X to get Y. Change this rule to get this outcome. And it seldom works.
Most people writing laws and policies are trying to achieve a particular goal. But they take what Megan Stevenson calls an engineer’s view, which “presumes a mechanistic structure that can be predictably manipulated to achieve social goals.” In their engineer’s view, policymakers miss the ways their rules will play out in a complex, adaptive system, rather than a complicated but fundamentally deterministic one. The lawmakers who outlawed or restricted abortion in their states after the Dobbs decision, for example, hoped that fewer women would have abortions. They may have succeeded locally, but nationally, abortion rates for 2023 are up over 2020.
Another example comes from civil service rules governing hiring. In theory, they combat nepotism and patronage. In practice, rules intended to avoid bias deter HR professionals from allowing domain experts to interview and assess candidates (nurses assessing candidates for nursing jobs, for example) or even using their own judgment. It’s safer to simply follow a prescribed set of procedures that are meant to appear objective but have become intricate, convoluted and at times nonsensical. That means that only candidates with inside knowledge of those procedures and rules can successfully navigate this highly specialized system. While the process is designed to discourage casual favoritism, it disadvantages qualified candidates who don’t have someone to champion their cause from the inside. We’re right back to “you have to know the right people.”
Likewise, later additions to civil service rules attempted to give preference to veterans in the selection process. But the way the veterans preference was actually implemented allows HR managers to apply it to a pool of candidates who have only passed a resume review and a self-assessment (succeeding in which, again, demonstrates competence in the highly specialized hiring process, not in the skills needed for the job). This poor process means that the hiring managers — those who will actually supervise these new employees — have been conditioned to assume that slates of veterans are unqualified, creating entrenched bias against veterans rather than a leg up.
Stevenson’s stabilizer effect can be seen operating here; interventions push toward an outcome, but over time, the culture (and human behavior) eats the intent of those policy interventions, and we are either back where we started — or sometimes worse off.
The “engineer’s view” leads into a delusion I have previously accused the tech world of indulging in, but now realize that policymakers are equally prone to: Death Star thinking. In the 1977 film “Star Wars,” the heroes acquire the plans for the Empire’s new space station designed to threaten the galaxy into submission, and they discover that if a talented fighter pilot could land one well-placed shot into the Death Star’s thermal exhaust port, it would cause the entire apparatus of their oppression to explode spectacularly. Stevenson would call this a cascade, and in fact, it seems the perfect visual metaphor. That shot is akin to social scientists’ holy grail: the “small, inexpensive intervention with large, widespread and lasting gains,” in Stevenson’s words. She calls the holy grail a myth, and perhaps George Lucas would agree: It turns out there are countless Death Stars ahead of us as the Imperial war (and the franchise) continues. Even that cascade is subject to the stabilizer effect in the longer run.
But the release that victory provided is irresistible, and has become the basis of the dominant theory of change of most policymakers. My work for the past ten years has been to explore the root causes of government’s general low competence and capacity in using digital technology to achieve its goals, and my diagnoses of the problem have led some lawmakers to assume I’ve got the blueprints of the Death Star and have found the vulnerability. They ask me, essentially, where to target the shot. There must be a change to procurement rules that would fix all this, or a way to increase accountability for digital leaders. Can’t we just mandate that all government websites be made as easy to use as Uber?
These legislators are doing the job they think they’ve been hired for: adding to or changing the rules of how government operates. In this regard, a different metaphor may be useful: They think of themselves as gardeners. If a bill they write becomes law, they’ve planted a seed, and they expect to gain status when it grows into something that provides shade, or fruit, or flowers for the public. But they don’t pay much attention to the seed after it’s planted, and today, many seeds fail to grow at all. Others grow but don’t provide the expected benefit. Of course, planting seeds is a tiny fraction of the work of gardening. Most of a gardener’s time is spent weeding and tending the soil, and lawmakers should invest similarly in repealing or replacing previous laws that haven’t worked out and making sure that the apparatus charged with implementing laws is healthy and capable. The general lack of interest in this caretaking work has resulted in an administrative state bogged down with decades of accumulated policy cruft and a civil service that’s overworked, underappreciated and hamstrung in myriad ways. The soil is desperately depleted, yet lawmakers still just want to know what seed to plant.
The fact that interventions mostly fail does not lead Stevenson to despair or hopelessness, but rather to a view that our efforts need to be directed at different kinds of work, including systemic reform, that are much harder to measure. Similarly, it’s not that I have no answers for our leaders looking to improve government’s competence: It’s that weeding and tending the soil is long, hard work that doesn’t provide an immediate payoff, is hard to sell to constituents and, yes, is hard to measure. As Stevenson says, “It’s not something you can do on your own; it requires changing the hearts and minds of large numbers of people, as well as changing the concrete structural factors of our lived experience.” Both her prescriptions and mine are also hard because the actors in these systems fail to imagine a world that is “otherwise the same as ours, while also being deeply, structurally different.”
But there is a framework for the kind of change I prescribe. Instead of managing administrative agencies through mandates and constraints, an enablement (soil-tending) approach seeks to build capacity by reducing or editing constraints (often in the form of clean-up of decades of accumulated policy cruft), and increasing the ability and willingness of the civil service to use appropriate judgment rather than hiding behind fidelity to process even when it results in bad outcomes. It particularly means resisting the urge to impose punitive measures on underperforming agencies, seeking to build rather than erode trust between the legislative and executive branches.
The changes Stevenson and I want to see are deeply connected. For all its flaws, social science nonetheless has an enormous impact on policy wonks, and a fundamental faith in the power of (mechanistic) interventions is shared between these worlds. If her provocation succeeds in opening up the world of social science to a more nuanced theory of change, that sensibility should over time influence policymakers toward enablement and capacity-building approaches. And if the public held their elected leaders accountable to the work of soil-tending, not just seed-planting, it could create demand among social scientists for different kinds of research. The building of state capacity both demands and enables the world Stevenson envisions.
 | A guest post by
|