4 min read
We Are Automating Judgment We Never Wrote Down
The hardest part of automating expert work is that the expertise was never written anywhere. When we hand that judgment to a model, we risk losing the very thing that made it good.
There is a kind of knowledge that lives only in people. A senior claims adjuster looks at a file and feels that something is off before she can tell you why. A veteran nurse knows which patient to check on first. A good editor reads a paragraph and knows it is wrong without reaching for a rule. None of this was ever written down, because it cannot be. It is the residue of ten thousand cases, and it exists as instinct rather than instruction.
We are now in the business of automating exactly this kind of work, and I do not think we have been honest about what that means. When a company decides to let a model handle the judgment that used to live in an expert, it is not copying the expert. It is replacing a thing it never understood with a thing it can at least see. That trade can be worth making. It is rarely examined before it is made.
The model learns the output, not the reasoning
When you train or prompt a model to do an expert's job, you give it examples of decisions. You do not give it the reasoning behind those decisions, because the expert could not fully articulate it either. The model gets very good at producing outputs that look like the expert's outputs. Whether it is reasoning the way the expert reasoned is something we mostly assume and almost never verify.
Most of the time this is fine, because most of the cases are routine and the routine cases are where the agreement between model and expert is strongest. The danger lives in the rare case, the one where the expert's instinct would have fired and the model's pattern matching does not. The expert could not have told you in advance which case that would be. That was the whole value of the expert. The model gives you confident, fluent output on that case too, and nothing in the output tells you it is the one that should have been escalated.
The expertise erodes once you stop using it
Here is the part that worries me more than any single wrong answer. Expert judgment is maintained by exercising it. The adjuster stays sharp by working files. The editor stays sharp by editing. When the routine work is handed to a model and the human is left to review only the model's output, the muscle that produced the judgment in the first place starts to atrophy.
Reviewing a model's decision is a different and weaker skill than making the decision yourself. It is easier to nod along to a fluent, confident answer than to construct one from scratch. So the people who were supposed to be the safety net gradually lose the very capability that made them a safety net. Five years in, the company has a model trained on the judgment of experts who no longer exist, reviewed by people who never developed the instinct, and no clear way to tell that anything has degraded.
This is not a hypothetical for fields with a long apprenticeship. The pipeline that produced the expert ran through years of doing the routine work badly and then less badly. If the routine work disappears, so does the path to expertise. We may be quietly dismantling the training ground for the next generation of the very people we are relying on to supervise the machines.
What this asks of us
I am not arguing against automating expert work. I have built systems that do it and I think they were the right call. I am arguing for honesty about the trade, and for a few specific habits that respect what is actually at stake.
Keep a meaningful share of the real work flowing through humans, not as a courtesy but as maintenance of a capability you will need. Treat the model's confidence as no signal at all about whether a case is the dangerous one, and build escalation paths that do not depend on the model recognizing its own limits. Pay attention to the cases the experts disagree about, because those are the ones where the model is most likely to be confidently wrong and where you can least afford to have lost the human who would have known.
Most of all, write down what you can while the experts are still here to ask. Not because you will capture the instinct, you will not, but because the act of trying reveals how much of the work was judgment you never priced. A company that understands what it is automating can make the trade deliberately. A company that thinks it is automating a documented process when it is actually automating an instinct is taking a risk it has not named.
The capability to automate judgment arrived faster than our wisdom about when to do it. Closing that gap is not a technical problem. It is a question of what we choose to keep doing ourselves, and we are answering it right now mostly by not asking it.
Let's Connect
8939 South Sepulveda Boulevard Suite 102
Los Angeles CA 90045
United States