Civic Innovations

Like many people, I’m in the process of transitioning my online activity away from Twitter and towards Mastodon (and more writing here). But a recent interaction with several people on Twitter around the use of algorithms in government programs, specifically public benefit determination, felt worthy of a deeper dive.

I tweeted the image above that riffs on a popular excerpt from a 1970’s era IBM training manual. The page from the IBM manual states that “A computer can never be held accountable. Therefore a computer must never make a management decision.” It’s a terrific encapsulation of the idea that offloading decision making to computers through artificial intelligence comes with potential accountability and ethical concerns.

I replaced the words “management decision” with “eligibility determination” because I think these concerns are even more acutely felt in areas where AI is used to replace human judgement in how people access public benefits. I ended up getting a fair bit of pushback from several people who obviously have more faith in the upside of AI than I do, and while I found myself disagreeing with them the conversation was pretty civil by Twitter standards. If you’re interested in the details, you can read through the thread.

Coming away from this exchange, I find myself thinking that this is an issue that is worth more thought and study by folks working in and around government. Governments making use of data and algorithms face a number of risks in ensuring that these tools are used effectively and responsibly. And I’m not convinced that we’re thinking about it enough.

Data quality is one important risk to consider: is the data accurate and complete? Does the data contain bias or errors that could skew results one way or another? Algorithmic accuracy is another important risk to consider — does the model that is being used produce accurate results?

It is also important to note that these are not new risks. Governments have faced these risks for as long as they have been using data and algorithms. Contemporary algorithm use, that is more closely tied to policy implementation and execution, does make these issues more acute, as the potential impact of bad data or algorithmic accuracy is more immediate. Fundamentally, these issues are the same as ones governments have faced for decades.

But there is a new kind of risk that contemporary algorithm use raises that is critical for governments to consider and address as they make use of these new tools. In the past, data and algorithms helped government policy makers understand the relationship between variables — education and wages, tax rates and the distribution of the tax burden. The outputs of these analyses helped inform decisions by policy makers or other actors in government.

Contemporary use of these tools is different in that algorithms now increasingly embody the capacity to make an inference or a judgement about something. In the past, we might have used data to try and identify the relationship between households with specific characteristics and the presence of at risk children to inform policy changes. Today, we use algorithms to identify which specific households government officials think have at risk children in them based on the data fed into a model. The model can infer that the risk of an at-risk child in a household is sufficient to warrant a government agency to take immediate action and intervene.

When we imbue the ability to make a judgement to an algorithm, a new class of ethical questions arises.

Who do we hold accountable for decisions that get made or actions that are taken by government when some part of the decision making has been ceded to an algorithm? And because the consequences of algorithm use in government are now much more immediate for specific individuals, how do we ensure that their use is being applied fairly and without prejudice or bias?

There are important discussions for current government employees, and those studying to become government administrators to have. It’s heartening to see more and more schools of public administration take up these issues and give students a grounding in contemporary data and algorithm use.

But as we arm government officials with new tools and training that could potentially revolutionize how we govern and provide public services, we should not lose sight of the ethical issues these new tools create.

We have more work to do.