Fix a mistake once
by turning every correction into a rule the agents read.

When Anthropic’s data team pointed Claude at its own warehouse without any written guidance, the agent answered business analytics questions correctly no more than 21% of the time on the team’s evals. With a set of written skills added, accuracy climbed to above 95% in aggregate. Today 95% of the company’s business analytics queries are automated, according to How Anthropic enables self-service data analytics with Claude, a June 3 post by five members of that team.

The part I keep rereading sits near the end of the post, in the plumbing. Every time a colleague corrects the agent in a thread, the team treats the correction as a candidate test. A scheduled agent scans the channels every few hours for correction language such as “that’s the wrong table,” drafts a one-line fix to the relevant reference document, and opens a pull request tagged to the person who owns that domain. The same corrections go into the test set. The authors call this “the part that closes the loop.”

Most companies correct their agents all day and keep none of it. The fix happens in a reply, the reply scrolls away, and next week somebody makes the same correction again. The asset that compounds is the log of those corrections, the rules written from them and the tests built on top, owned by named people in the business. It is also the one part of an AI setup that survives the next change of model.

2195%

Accuracy of Claude on analytics questions without and with written skills, per Anthropic’s data team, June 2026.

90%

Share of that team’s data-model changes that now update a skill file in the same change.

20–50

Test cases drawn from real failures that Anthropic’s engineers call “a great start” for an eval suite.

The jump came from written corrections

The post credits the difference between 21% and 95% to written material: definitions of what each metric means, which tables to trust, which filters always apply, and what a finished analysis looks like. Much of it exists because someone got a wrong answer, said so, and the fix was written down where the agent reads it.

The team treated this as engineering work. The skill files live in the same repository as the data models, so the change that alters a table is the same change that updates the document describing it. A domain owner cannot announce the agent to their colleagues until their slice of the test set passes a threshold, which the team set at about 90% at first. Every run is stored with the model version and skill version, so “did that change help?” becomes a query.

One finding should interest anyone tempted to let the machine write its own rulebook. The team tried having a model generate metric definitions automatically from raw tables and query logs. The result “encoded the very ambiguities we were trying to eliminate” and did worse on their tests than a smaller set written by people. Their recommendation is to let Claude draft the documentation while “having a human own the definition.”

A correction left in a chat gets made again

The same habit shows up wherever agents do daily work. Boris Cherny, who leads Claude Code at Anthropic, described his team’s setup in January: one shared instruction file, checked into version control, that the whole team edits several times a week. “Anytime we see Claude do something incorrectly we add it to the CLAUDE.md, so Claude knows not to do it next time.” Other teams keep their own file, and “it is each team’s job to keep theirs up to date.”

That is the whole mechanism, and it is cheap. It needs a habit and an owner. Without both, each correction costs a few minutes of a senior person’s time, the agent never learns from it, and the person correcting slowly concludes the agent is not worth the trouble.

Writing the rule down is half the job. Kasey Klimes, who builds decision software at Primitive and so has a product in this space, put the gap well in August: “memory is half the equation. a company brain needs reliable enforcement of the decisions it has made.” A rule in a file is a request. A test that fails when the agent breaks the rule is enforcement.

Tests turn a rule into something that outlives the model

An eval is a test for an AI system: give it an input, check the output against an answer someone has agreed is right. Anthropic’s engineering team published a guide to evals for agents in January, and its advice for where to start is plain: look at the bug tracker and the support queue, because “converting user-reported failures into test cases ensures your suite reflects actual usage.” Tests that check whether the agent still handles everything it used to “should have a nearly 100% pass rate.” On who writes them: “The people closest to product requirements and users are best positioned to define success.”

The guide also explains why this matters to a budget. “When more powerful models come out, teams without evals face weeks of testing while competitors with evals can quickly determine the model’s strengths, tune their prompts, and upgrade in days.” A suite built from your own corrections tells you within days whether a new or cheaper model is good enough.

Satya Nadella turned the same idea into a test of ownership in a long post in June.

A company should be able to switch out a “generalist” model without losing the “company veteran” expertise built into their learning system. This is the key “test” of your control and sovereignty in the era ahead.

In the same post he writes that “private evals should capture whether a model is actually improving against outcomes that matter to the business (not just external benchmarks!).” Nadella runs a company that sells the platform this loop would sit on, so read it as a vendor’s position as well as a strategy. Garrett Lord, CEO of Handshake, quoted him a week later and asked “how many business leaders can actually quantify how accurate their AI programs are?” Handshake sells evals, so that is a seller’s question, and still a fair one to put to your own team.

I turn Nadella’s test into a budget rule. Go down the AI spend line by line and ask what would be left if the model changed tomorrow. Licenses, prompts tuned to one model’s quirks and fine-tunes tied to one model generation would mostly have to be redone, so I treat them as rent. The correction log, the rules written from it and the tests built on them carry over to whatever model comes next, so I treat them as the asset. In most companies the asset column is close to empty, and nobody is named as its owner.

The method: from correction to rule to test

This is the loop we set up, one process at a time. It needs a folder, a habit and a name to start.

1. Capture every correction in one place. When someone fixes an agent’s output, they log it in a shared list: what the agent produced, what it should have produced, and why. One line each. If corrections happen in chat, a weekly sweep by hand works until an agent can scan for them.
2. Turn the correction into a rule. The owner writes one plain sentence the agent can follow, such as “Freight from supplier X is always billed on the second invoice, so never flag it as missing on the first.” Each rule gets an ID, a date and the correction it came from.
3. Store the rules where the agent reads them. One plain text file per process, kept in a versioned folder the company controls, loaded by the agent before every run. The vendor’s settings screen is the wrong place, because the rules then leave with the vendor.
4. Write the test from the corrected example. The original input and the corrected output become a test case. Start with the 20 to 50 cases from last month’s corrections, and rerun the set on every change of model, rule or data source.
5. Give each process a named owner in the business. The controller owns the close rules and tests. The sales operations lead owns the quoting rules and tests. The owner approves new rules, removes stale ones and signs off before a new model goes live, the same way someone owns a general ledger account.

The owner is the step companies skip, and it is the one that decides whether the rest holds. Only the person who knows why freight from supplier X is billed late can say whether a rule is right. That also gives auditors a clean trail of how the agent was supervised, which fits the controls I set out in an earlier note on agents in the ledger.

Track one number: the repeat-mistake rate

Each month, count the corrections logged for a process and mark the ones that repeat a mistake the agent has made before. The repeat-mistake rate is the share of corrections that are repeats. If it falls, the loop works. If it stays flat while the agent runs more jobs, people are correcting by hand and nothing is being kept.

The repeats split in two, and each half points at a different fix. A repeat with no rule behind it is a capture failure: someone corrected the agent and nobody wrote it down. A repeat that breaks a rule already on file is an enforcement failure: the rule is unclear, the agent is not reading it, or it needs a test that fails loudly. I would put both counts in the monthly operations review next to the process’s volume and error figures. If you do not yet know what the process looked like before the agent, start with the baseline first.

Rules go stale, so someone has to delete them

A log that only grows becomes its own problem. Anthropic’s data team writes that it regularly prunes skill scaffolding “as models improve and previous failure modes no longer apply.” A rule written for last year’s model can block the better behavior of this year’s, and a rule written around a broken system keeps the workaround alive after the system is fixed. The owner’s job includes a quarterly pass to remove rules whose test now passes without them.

The loop also misses some mistakes. The Anthropic post is candid that “the failure mode none of this fully catches is the silent one,” a wrong answer that looks plausible and is used without objection. Its mitigations include human sign-off on anything going to leadership. Corrections only exist where someone checks, so the processes with no reviewer will show a low repeat rate for the wrong reason. Recording why people made a decision, which I covered in a note on what a company knows, is a separate habit. This loop works on the agent’s daily output.

What a COO can do in the next 30 days

Pick one process where an agent already produces work that people check, such as invoice coding, quote drafting or the first pass of the month-end reconciliation. Name one owner from the business side. Create the rules file and the correction log, and ask everyone who reviews the agent’s output to log each fix for four weeks.

At the end of the month, have the owner turn the corrections into rules and the first 20 to 50 test cases, and compute the repeat-mistake rate for the month. Put that number on the agenda of the next operations review. Before the next AI contract renewal, add one clause: the rules, the correction log and the tests are stored in plain files the company owns and can take to another model.