A Management Problem in a Technology Costume

Six months ago I sat in a review meeting where a delivery lead explained, with real conviction, why the AI rollout on her team had "failed." She had the deck. Adoption was down to a third of what it had been in week two. Two of her best analysts had quietly stopped using the tool and gone back to the old way. The vendor's numbers looked nothing like the pilot's numbers. Everyone in the room nodded along to the obvious diagnosis: the model wasn't good enough for the work.

I want to be honest about what I actually think happened, because I was close enough to this one to watch it in real time, and it wasn't that. The model was, if anything, slightly better by week twelve than it had been at the pilot — the vendor had shipped two updates in between. What had changed was who was allowed to disagree with it, and nobody in that room had noticed that this was the actual variable they were managing.

This is the pattern I now see everywhere I look for it, across a services organisation I've spent the better part of two years embedded with, in various rollouts, various teams, various tools. Every AI rollout that visibly failed, failed on the org chart, not on the model. The technology is the costume. The management failure underneath it is almost always the same shape, and it is almost never named as a management failure, because "the AI didn't work" is a much more comfortable sentence to say out loud than "we didn't decide who gets to overrule it."

The rollout that "failed"

Here is what actually happened on that team, reconstructed from the retro notes and three separate conversations afterward, with names and identifying detail stripped out because the lesson is what matters, not who made the mistake.

The tool was a drafting assistant for a category of client-facing documents — the kind of work that used to take a mid-level analyst most of a day and now took the model about ninety seconds to produce a first pass. The pilot, run with four volunteers who had asked to be on it, was a genuine success: turnaround time down by two-thirds, quality holding, everyone involved enthusiastic. Leadership did the reasonable thing and rolled it out to the full team of twenty-two.

What nobody decided, in the move from four volunteers to twenty-two conscripts, was who had the standing to say "the draft is wrong, redo it by hand." The four pilot users were senior enough, and had enough political capital on the team, that overruling the model was simply something they did without asking anyone. The other eighteen did not have that standing. When the model's draft was subtly wrong — right structure, wrong emphasis, a client-specific nuance it had no way to know — the senior four caught it and fixed it in minutes. The other eighteen either shipped it as-is, because questioning the tool felt like questioning their own competence to their manager, or they quietly abandoned it and went back to writing from scratch, because that felt safer than being the junior person who "couldn't get the AI to work."

"Nobody told the team it was safe to overrule the model. So the ones with the least standing to overrule anything just stopped using it."

That is the entire failure. Not the drafts. The permission structure around the drafts.

The org chart is the real architecture

I keep coming back to a phrase from an internal note written by one of the four pilot volunteers, trying to explain to her manager why the wider rollout wasn't reproducing the pilot's numbers: "it's not that the tool is different for them. It's that they are different from us, in exactly the one dimension that matters here." She meant seniority — but more precisely, she meant the standing to be believed when you say the machine is wrong.

Every services organisation I've worked inside runs on a chain of trust that long predates any AI tool: a junior person's work is checked by a senior person, whose work is checked by a partner, whose name is on the client relationship. That chain is not incidental to how the firm operates — it is how the firm operates. Client trust flows down through it and accountability flows back up through it, and everyone in the chain knows, without being told, roughly how much independent judgment they're allowed to exercise at their level.

An AI tool that produces work of senior-analyst quality, handed to a junior analyst, breaks that chain in a way nobody planned for. The junior analyst is now holding output that looks better than anything they could produce themselves, from a source that isn't part of the chain of trust at all. They have no established standing to evaluate it, override it, or defend a decision to change it. So they default to the two available behaviours: rubber-stamp it, or route around it. Neither is what leadership wanted, and neither shows up anywhere on a model-quality dashboard.

What broke, precisely

It's worth being specific about the mechanism, because "trust" is the kind of word that sounds true and explains nothing on its own. Three things broke, in order.

Read that list again and notice that not one item mentions the model. This is why "the AI wasn't good enough" is such a seductive, wrong explanation — it points at the one component of the system that's actually easiest to inspect and hardest to be personally responsible for. Pointing at the org chart means pointing at a decision leadership made, or more often, a decision leadership didn't make and should have.

The dashboard that was measuring the wrong layer

I want to dwell on the measurement failure for a moment, because it's the part I find most instructive and the part most leaders skip past on their way to the org-chart lesson. Leadership on the failed rollout was not asleep at the wheel. They had a dashboard. It tracked draft turnaround time, a sampled quality score run weekly against a rubric, and adoption — the percentage of eligible documents that went through the tool rather than being written from scratch. By every one of those three numbers, the rollout looked fine for the first six weeks. Turnaround was down. Quality scores held. Adoption was north of eighty percent.

What the dashboard could not see, because nobody had built a metric for it, was the distribution of override behaviour underneath that eighty percent. Aggregate adoption of eighty percent is consistent with two very different realities: a team confidently using the tool and occasionally correcting it, or a team rubber-stamping the tool's output because correcting it doesn't feel safe. Those two realities produce identical adoption numbers and wildly different downstream risk, and the difference between them is invisible to any metric that only counts whether the tool was used, not how the human treated its output afterward.

By the time the quality score dipped — around week eight, when a subtly wrong draft from the tool made it all the way to a client without anyone catching the error — leadership had six weeks of green dashboard behind them and no instrumentation that could explain why. The natural, wrong conclusion was that the model had degraded. The actual explanation, visible only in hindsight and only by reading the team's internal chat history, was that the rubber-stamping had been quietly accumulating the entire time, underneath a metric that had no way to register it.

"Adoption and quality dashboards measure whether the tool was used and whether its output was good. Neither one measures whether the human treating that output felt safe enough to argue with it. That third number is the one that actually predicts failure, and almost nobody builds it."

This is, I think, the sharpest version of the argument for why this is a management problem rather than a technology one. A genuinely broken model shows up in a quality score within days. A broken permission structure can hide behind a healthy quality score for months, because the people absorbing the risk of that broken structure are, by construction, the people with the least standing to flag it. The junior analyst who rubber-stamps a wrong draft rather than argue with it is not lying to the dashboard. She is behaving exactly as the unspoken incentives told her to behave, and the dashboard was never built to see that behaviour in the first place.

The costume comes off

The team that got this right — a different group, same firm, six months later, running the same category of tool for a different document type — did one thing differently, and it wasn't a better prompt library. They spent the first two weeks of the rollout explicitly assigning override authority. Every analyst, at every level, was told in writing: you are expected to change the model's output when you believe it's wrong, this is not a mark against you, and if you're not sure whether a change is warranted, that specific uncertainty goes into a shared channel where a senior person answers it within the hour.

The effect wasn't subtle. Override rates in week one were high — junior staff, given explicit permission, second-guessed the model constantly, sometimes unnecessarily. By week four the rate had settled into something close to what the senior pilot users had shown from day one: confident acceptance of the routine cases, fast escalation of the genuinely uncertain ones. What had changed was not the model and not the staff's underlying skill. It was that the chain of trust had been rebuilt on purpose, instead of being left to reassemble itself informally around whoever happened to have standing already.

"We didn't train the team to use the tool better. We told them, explicitly, who was allowed to say it was wrong. That was the whole intervention."

What actually changed

I don't think this is a story about AI specifically, and I think that's the useful part of it, not a caveat. Every technology that changes who can produce senior-quality output changes the org chart underneath it, whether or not anyone updates the org chart on purpose. Spreadsheets did this to bookkeeping. Search did this to research. What's different about this wave is the speed — a team can go from "four enthusiastic volunteers" to "twenty-two conscripts with no updated permission structure" in a single leadership decision, made in a single meeting, with genuinely good intentions and zero malice, and the failure that follows will still look, on every available dashboard, like a technology problem.

It isn't. It's a management problem that happened to be standing next to a model when it went wrong, and the costume is doing a lot of work to keep leadership from noticing whose decision actually needs revisiting. The fix, when it works, is almost never a better tool. It's someone in the room saying, out loud and in writing, who gets to overrule it — before the rollout, not after the retro.

I've started asking a single question in every rollout planning meeting I sit in on now, before a single vendor slide is shown: when this tool hands someone work that's better than they could produce alone, who has standing to tell it it's wrong, and how will they know that standing is theirs? It is a slower question than "which model do we license," and it is, in every case I've watched, the one that actually determines whether the rollout works. The technology decision usually takes an afternoon. Answering that question properly takes the two weeks nobody wants to spend on it, and skipping it is how a management failure ends up filed, six months later, as a story about the limits of AI.