5 min read
The Cost of Letting a Model Decide
When a model makes a decision that affects a real person, someone is accountable for it. Too many companies are deploying systems that decide without ever answering the question of who that someone is.
A model denies someone a loan. A model flags an account for review and freezes it. A model ranks a job applicant low enough that a human never sees the file. Each of these is a decision that lands on a real person, and each of them is increasingly being made by a system that nobody in the company can fully explain. The question I keep asking clients, and the one too few of them have an answer to, is simple. When this goes wrong, who is accountable.
The reason the question matters is that accountability does not disappear when you automate a decision. It just gets harder to locate. The person affected still suffers a real consequence. The company still owns the outcome. What changes is that the chain from decision to decider now runs through a model whose reasoning is opaque, and a lot of organizations are using that opacity, intentionally or not, as a place for responsibility to go and quietly vanish.
Automation does not transfer the blame to the machine
There is a comfortable fiction that once a model makes a decision, the decision becomes a kind of natural fact, the way a thermometer reports a temperature. The model said no. As if that settles it. But the model did not decide to deploy itself, did not choose the data it learned from, did not set the threshold that turned a score into a denial. People made all of those choices, and the consequences of the model's decisions trace directly back to them.
When a company hides behind the model, it is not actually removing the accountability. It is just making it harder for the affected person to reach. The applicant who was filtered out cannot interrogate a model. They could have questioned a human. The automation did not make the decision more fair or more correct. It made it less answerable, and less answerable is a real harm even when the decision happens to be right.
Opacity is a choice, not a property
People talk about model decisions being unexplainable as though it were a law of nature. It is mostly a choice about how much you were willing to invest in being able to explain. You can log every input to a decision. You can record what the model saw and what it produced. You can build the system so that any decision can be reconstructed and reviewed rather than shrugged at. We do this on the systems we build, not because it is required but because a decision you cannot reconstruct is a decision you cannot defend, to a regulator, to a customer, or to yourself.
A company that deploys a deciding system with no way to explain any individual decision has chosen not to be able to answer for it. That choice is usually invisible at deployment, when everything works, and becomes very visible the first time a decision is challenged and the honest answer is that nobody knows why the model did what it did. By then the cost of having skipped the explainability work is much higher than building it in would have been.
The decisions that should not be fully automated yet
Not every decision carries the same weight, and the mistake is treating them as if they do. A model that routes a support ticket to the wrong queue costs minutes and gets corrected. A model that denies someone credit or housing or a job changes a life, and the asymmetry between those two cases should change how much human judgment stays in the loop. The cheap, reversible, low stakes decision is a fine candidate for full automation. The expensive, irreversible, life affecting one is not, no matter how accurate the model looks in testing.
The honest line is drawn by the cost of being wrong to the person on the receiving end, not by the accuracy of the model on average. A system that is right ninety nine percent of the time is still catastrophically wrong for the one person in a hundred whose life it derails, and average accuracy offers that person no comfort at all. For decisions where a wrong answer is that costly to a real human, the model should inform a person who decides and remains accountable, rather than deciding alone.
What accountable automation requires
A company that wants to automate decisions responsibly can do it, and the requirements are not mysterious. Know, before deployment, who owns the outcomes of this system and can answer for them. Build the logging that lets any individual decision be reconstructed and explained after the fact. Keep a human in the loop for decisions whose cost of error to a person is high, and make sure that human has real authority to override rather than a rubber stamp. Give the affected person a route to challenge a decision and reach someone who can actually examine it.
None of this is exotic and most of it is the same governance any consequential process has always needed. The difference is that automation makes it easy to skip, because the system runs fine without it right up until the day it does not. The companies that will weather the coming scrutiny of automated decisions are the ones building the accountability in now, while it is cheap and optional, rather than after a regulator or a lawsuit makes it expensive and mandatory.
Letting a model decide is not free even when the model is good. The cost is the accountability you have to keep holding, deliberately, instead of letting it slip into the machine where no one can reach it. A company that pays that cost on purpose can automate with a clear conscience. A company that pretends the cost went away is borrowing against a debt that always, eventually, comes due.
Let's Connect
8939 South Sepulveda Boulevard Suite 102
Los Angeles CA 90045
United States