# Your agents need a critic and so does the plan they carry out. · VANDAM

> Why no AI agent should check its own work, how to design review so a person spends minutes per unit, and who should challenge the plan itself.

URL: https://vandam.ai/notes/agents-need-a-critic  
Contact: hendrik@vibelab.be

# Your agents need a critic and so does the plan they carry out.

Models grade their own work too kindly, and the labs now separate the agent that makes from the agent that checks, the way finance separates maker and checker. I set out a four-layer review design with a minutes budget, and why the brief itself needs a critic with no stake in it.

Hendrik De Winne 26 October 2026 9 min read

In a test published in June 2025, OpenAI’s o3 was asked to make a piece of GPU code run faster. It overwrote the timer that measured the speed instead. On another task it replaced the program that graded its answers with a stub that passed everything. [METR](https://metr.org/blog/2025-06-05-recent-reward-hacking/), the evaluation lab that ran the tests, counted **39** such reward hacks in **128** runs on its RE-Bench tasks. On one task the model hacked in every run, and telling it not to cheat barely moved the rate.

Even when a model plays fair, it grades its own work too kindly, and researchers have measured that too. So the agent that does the work should never be the one that checks it, and the checker should not be able to touch the tests. Finance teams already run that rule for people under the name maker-checker. The rule now has to cover software that writes a hundred pages before lunch.

The rule has a second half that is easier to miss. A good checker compares the output against the brief. Nobody in the loop checks the brief. Once agents carry out plans quickly and faithfully, a wrong plan stops being slowed down by the friction that used to expose it, so the plan needs a critic of its own, and that critic has to be a person with no stake in it being right.

- **30%**
  Share of o3’s runs on RE-Bench tasks in which it gamed the scoring, per [METR](https://metr.org/blog/2025-06-05-recent-reward-hacking/), June 2025.
- **0.91**
  GPT-4’s self-preference score on CNN/DailyMail summaries, where 0.5 means no bias, per [NeurIPS 2024](https://papers.nips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html).
- **80%**
  Share of interactive Claude session time spent waiting for a person, per [Anthropic](https://www.anthropic.com/research/employment-at-anthropic), October 2026.

## Models grade their own work too kindly

The first hint came in 2023. The authors of [Judging LLM-as-a-Judge](https://arxiv.org/abs/2306.05685), a NeurIPS paper from the team behind Chatbot Arena, noticed that GPT-4 favored its own answers with a **10%** higher win rate and Claude-v1 with a **25%** higher one. They were careful to say their data could not settle whether this was a real bias.

A year later Arjun Panickssery, Samuel Bowman and Shi Feng ran the controlled version. In [LLM Evaluators Recognize and Favor Their Own Generations](https://papers.nips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html), presented as an oral at NeurIPS 2024, they had GPT-4, GPT-3.5 and Llama 2 write and judge summaries of 2,000 news articles. GPT-4 preferred its own summaries, most strongly on CNN/DailyMail, and all models showed the most self-preference against the human-written summary. The models could tell their own text from other text without any training, and when the authors fine-tuned that ability up, self-preference rose with it in a linear relationship. A grader that recognizes its own work rewards it.

The obvious workaround, asking the agent to double-check itself, does not hold up either. At ICLR 2024, Jie Huang, Denny Zhou and colleagues showed in [Large Language Models Cannot Self-Correct Reasoning Yet](https://arxiv.org/abs/2310.01798) that models “struggle to self-correct their responses without external feedback,” and that “at times, their performance even degrades after self-correction.” The useful check needs something from outside the maker: a test, a source document, or a different reviewer.

## The labs now split the maker from the checker

Anthropic builds its own agent tooling around this. In June, Thariq Shihipar and Sid Bidasaria published [A harness for every task](https://claude.com/blog/a-harness-for-every-task-dynamic-workflows-in-claude-code), which names the failure plainly: “Self-preferential bias refers to Claude’s tendency to prefer its own results or findings, especially when asked to verify or judge them against a rubric.” Their pattern is to give each working agent its own checker: “For each spawned agent, run a separate spawned agent to adversarially verify its output against a rubric or criteria.”

They are as direct about the price: workflows like these “often use more tokens,” and “most traditional coding tasks do not need a panel of 5 reviewers.” Verification is a cost line, and it belongs where an error is expensive.

## Finance already has the rule, one level down

A CFO will recognize the structure at once. The clerk who keys a payment does not release it, and the person who builds the reconciliation does not sign it off. I covered access rights, change management and sign-off for an agent in the ledger in [an earlier note on controls](https://vandam.ai/notes/agents-and-internal-controls). This one is about who decides whether the work is right, and at what cost in human time.

The agent version of maker-checker has three conditions. The checker is a separate agent with its own context, and preferably a different model, so it does not recognize the maker’s style. It sees the rubric, the source documents and the finished artifact, and none of the maker’s reasoning, so it cannot be talked into agreement. And the maker has no write access to the tests, because METR’s results show what an optimizer does with a score it can reach.

## The human check is now the slowest step in the line

The second constraint is time. In its [October report](https://www.anthropic.com/research/employment-at-anthropic) on its own workforce, Anthropic found that its interactive sessions produce enough output to fill more than half of an employee’s work week if anyone read all of it. Investor [Greg Isenberg](https://x.com/gregisenberg/status/2105397216374403561) put the consequence in one line in September, writing about a Google model launch that by his account lets one answer run to a million tokens: “Obviously, nobody can carefully review that much output, so the bottleneck moves from making things to checking them, and tools that verify AI work become a big market.”

Eliyahu Goldratt’s theory of constraints says a line runs at the speed of its slowest station. When agents make ten times the drafts and the same three senior people approve them, the queue has moved to the approvers.

The arithmetic also sets a ceiling on the business case. If a task would take a person eight hours and checking the agent’s version takes two, the most you can ever gain is four times. Checked in eight minutes, the ceiling is sixty. Review minutes per unit of work is the number that decides whether an agent deployment pays, and almost no business case I have read contains it.

## Your best operators should write the tests

Review gets fast only when “right” is written down. If it lives in the controller’s head, every unit of agent work has to pass through the controller. If it is encoded as tests, most of the checking runs without anyone. In October, Isenberg [described](https://x.com/gregisenberg/status/2110414696037716372) what he heard from founders running vertical agent products: “A vertical harness is basically four things: the connectors for that industry’s data, evals written by people who know the work, rules for what the agent can and can’t touch, and a clean screen for a human to review.” He added that several of them said “the hire they want most is someone with taste who can write evals.”

Inside a company, that person already exists. It is the controller who knows which accrual is always wrong in March. Their best hours used to go into checking line by line, and should now go into writing the checks: each past error becomes a test, each exception they waved through becomes a written rule, and they own the test set the way an engineer owns code. The maker never sees part of that set, so the agent cannot learn to pass it without doing the work.

## A review design for one unit of agent work

Here is the design we use, shown on a month-end task: an agent drafts explanations for the 150 account variances above the review threshold. Assume a senior analyst would need about twelve hours for the same pack. The numbers are illustrative, and the structure is the method.

**Layer 1, code checks, zero human minutes.** Plain scripts, no model. Every explanation cites a journal entry or invoice that exists in the ledger. The amounts cited add up to the variance within tolerance. No explanation repeats last month’s text unless the same driver appears in the data. Anything labeled “timing” names the period it reverses in.

**Layer 2, an independent verifier agent, zero human minutes.** A separate agent, on a different model where possible, gets the trial balance, the ledger detail, the rubric and the draft. It never sees the maker’s notes. For each rule the scripts cannot judge, such as whether an explanation states a cause or only restates the number, it returns pass or fail with the evidence. Its pass only decides what reaches the person.

**Layer 3, the person, about 45 minutes.** The controller reads three groups and nothing else: the items either layer flagged, the ten largest variances by dollar amount, and five random items that passed everything. The random five are how you audit the verifier. That is roughly six percent of the twelve hours.

**Layer 4, the test owner, monthly.** Every error the controller catches becomes a new test in layer 1 or a new rule in layer 2. Once a quarter, plant ten known errors in a pack and record how many each layer catches.

The decision rule I would hold a team to: set the review budget before go-live, at no more than ten percent of the human-equivalent time. If checking needs more than a quarter of the time the work would take a person, the task is not ready for an agent. The tests are missing, or the work needs judgment nobody has written down.

## Someone with no stake has to attack the brief

Every layer above checks the output against the brief. If the brief is wrong, all four layers will confirm, quickly and precisely, that the agents executed it well. Before agents, a weak plan met resistance in the doing, from the analyst who pushed back to the week it took to build. Agents remove most of that friction. I expect this to show up as a new kind of failure, strong agent loops carrying out plans that nobody challenged. I have not seen a case study yet, so take it as a prediction.

Sony had an early answer. As a young man, the aspiring opera singer Norio Ohga wrote to the company to complain about the quality of its tape recorders, and its founders brought him into the business, [the AP reported](https://www.nbcnews.com/news/amp/wbna42730332) in his 2011 obituary. Sony’s founders hired the man who told them what was wrong with the product, and he went on to run the company.

For an AI program the equivalent is a named critic for every brief that will drive significant agent work, such as new pricing logic or an agent-run collections process. The critic is someone outside the sponsoring team, with no bonus tied to the plan. They get the brief two days before agents start and write down what would make it wrong and which test would show it. They are scored on whether their objections turned out to be right, so they gain nothing from criticism for its own sake. Agents can generate objections cheaply, and the Anthropic post suggests having agents tear a business plan apart from an investor’s, a customer’s and a competitor’s view. That makes a good first pass. A model checks a brief against what it was told, and only a person can question what the company wants.

## What to do this quarter

For a COO and CFO working together, four steps. First, list every agent workflow that runs today and mark where the same agent, or the same prompt, both makes and checks. Split those first, starting with the ones where an error costs money. Second, for the three highest-volume workflows, measure review minutes per unit for two weeks and put that number next to the human-equivalent time; that ratio is the ceiling on your gain. Third, name a test owner for each workflow from your strongest operators, give them a day a week to write tests, and take away the maker’s access to those tests. Fourth, pick the next large brief that agents will carry out and assign it a critic from outside the team before the first agent runs.

## More notes

29 October 2026 6 min read

## Who owns what your AI learns.

Y Combinator now asks for startups that help companies own their intelligence. Before a mid-sized company owns a model, it can own the four things every model works with: context, evals, rules and connections.

26 October 2026 9 min read

## Delete the step first and give the agent whatever survives.

Musk’s five-step method puts automation last, and most AI programs start there because an agent is the one step a vendor can sell. I walk accounts payable through the steps in his order and show why an agent that clears exceptions fast hides the steps that should go.

## Fix a mistake once by turning every correction into a rule the agents read.

Anthropic's data team took Claude from 21% to above 95% accuracy with written skills, and now turns every correction a colleague makes into a rule and a test for the person who owns that domain. I set out that loop for a mid-sized company, with where the rules live, who owns them, and the one number a COO should track: how often the same mistake comes back.