Judgment Over Prompts

A hiring manager on the team I've been embedded with asked me, about a year ago, whether she should add "prompt engineering experience" as a line item on a job posting for a new analyst role. I told her not to bother, and I want to walk through why, because the reasoning turned out to matter for a lot more than that one posting.

The skill that had a six-month half-life

We'd hired, briefly, for exactly that skill about eighteen months earlier — someone whose main qualification was an unusually deep, almost obsessive fluency with prompt structuring, few-shot examples, chain-of-thought scaffolding, the whole toolkit. He was genuinely excellent at it, and for about two months his output was noticeably better than everyone else's on the team. Then two things happened at once. The underlying model got an update that made a lot of his careful scaffolding redundant — the new version just didn't need the same coaxing to produce good structure. And the rest of the team, watching him work, picked up the basics of what he was doing within a few weeks, because prompting technique, it turns out, is not that deep a well. By month six, his prompts looked like everyone else's prompts, and his output quality advantage had evaporated with them.

What hadn't evaporated, and what actually separated the strong performers from the average ones a year later, was something else entirely: how well each person could look at a model's output and know, quickly and specifically, what was wrong with it and what to do about it. That skill didn't commoditise the way prompting did, because it wasn't really about the model at all. It was the same editorial judgment a good editor brings to a junior writer's draft — a skill that predates AI by decades and that AI, if anything, made more valuable rather than less.

"The prompt-engineering advantage lasted about two months. The editorial-judgment advantage is still there a year later, and it isn't shrinking."

What we actually started training for

Once we noticed this, the hiring and onboarding process changed in a specific, concrete way. Instead of screening for prompting cleverness, we built an exercise around output diagnosis. Candidates get three pieces of model-generated work on a realistic task — a client memo, a data summary, a process description — and the job is not to improve the prompt that produced them. It's to mark up each one the way a demanding editor would mark up a junior colleague's draft: what's actually wrong here, why, and what would you tell the person who wrote this to do differently next time.

The candidates who did well on this exercise weren't necessarily the ones with the most AI experience. They were, overwhelmingly, the ones with strong general editorial instincts — people who'd spent time as writers, as reviewers, as the person on a team whose job was to catch what was subtly off in someone else's work before it went out the door. That instinct transfers almost directly onto AI output, because the failure modes are structurally similar: plausible-sounding, well-organised, and occasionally, quietly wrong in a way that only someone who actually understands the underlying subject would catch.

The interview question that replaced the prompting test

We also killed the prompting test itself, which had been a fixture of the interview loop for the better part of a year: give the candidate a task, watch them write a prompt, score the cleverness of the structure. In hindsight it was measuring almost exactly the wrong thing, because a clever prompt and a well-judged final answer turned out to be only loosely related — some of the best prompters we'd hired produced output that read beautifully and was wrong in ways they never caught, because all their effort had gone into the question rather than the scrutiny of the answer.

The replacement question is blunter and, I think, more honest about what the job actually is. We show the candidate a real model output on a realistic task and ask a single question: "if you had to put your name on this and send it to a client in the next ten minutes, what, if anything, would you change first — and how would you know?" There's no prompt to admire and nowhere to hide behind technique. The candidate either has a specific, defensible answer, or they don't, and the ones who don't are the ones who, historically, struggled most once they were actually doing the job.

"The old test asked candidates to write a good question. The new one asks them to catch a bad answer. Only one of those is what the job actually requires all day, every day."

The onboarding change that mattered most

The single highest-leverage change we made to onboarding wasn't a prompting tutorial at all — we cut that down to about twenty minutes, because there just isn't that much to teach. It was pairing every new hire with a senior reviewer for their first month of AI-assisted work, specifically to narrate, out loud, why they were accepting or rejecting each piece of model output. Not "here's a good prompt template." Instead: "here's why I don't trust this number, here's why this paragraph's structure is actually fine even though it reads a little stiff, here's the one phrase in this draft that tells me the model didn't understand the client relationship."

"We stopped teaching people how to ask better questions. We started teaching them how to read the answers like an editor would — because that's the skill that was actually scarce."

That narration is slow to deliver and expensive in senior reviewer time, and it is, six months in, the highest-return training investment the team has made on this front. New hires who went through it hit competent, independent judgment on AI output in roughly six weeks. New hires who only got the old prompting tutorial took closer to four months to reach the same bar, and some of them, we suspect, never fully closed the gap — because nobody had ever explicitly told them that the skill they needed wasn't the one the tutorial covered.

I don't think this is a controversial claim once you sit with it, but it runs directly against how most firms are still staffing for this shift — job postings asking for "AI fluency," training budgets going almost entirely to tool tutorials, performance reviews that credit people for adoption rates rather than catch rates. All of that optimises for the skill with the shortest half-life in the building. The judgment to know what to do with a fluent, confident, occasionally wrong answer is the actual scarce resource, and it was scarce before any of this started. AI didn't create the need for it. It just made the absence of it visible faster, and more expensively, than almost anything that came before.

The hiring manager who asked me the original question ended up rewriting the posting entirely. The line about "prompt engineering experience" came out. In its place went a single sentence that took her longer to write than the rest of the listing combined: "we're looking for someone who's comfortable telling a confident, well-written answer that it's wrong." She told me afterward that it was the first job posting she'd ever written that made a candidate visibly pause in the interview before answering — which, on reflection, is exactly the reaction the question is supposed to produce. The people who can sit with that pause and still give a specific answer are the ones worth training. Everyone else was only ever going to be as good as their last prompt, and their last prompt was never going to be the moat.