# Most of your AI list needs software and only a few items need an agent. · VANDAM > Most AI requests are rules for software. Agent reliability and cost data, four tests, and ten mid-market requests sorted into software, agent or person. URL: https://vandam.ai/notes/most-of-it-needs-software Contact: hendrik@vibelab.be # Most of your AI list needs software and only a few items need an agent. Agents still finish a minority of office tasks and give different answers on repeated runs, while a written rule does the same thing every time for almost nothing. I set out four questions that sort every request into software, agent or person, and apply them to ten typical ones. Hendrik De Winne 26 October 2026 9 min read Researchers at Carnegie Mellon built a mock software company, with a chat system, a file server and simulated colleagues, and gave AI agents 175 ordinary office tasks to finish on their own. In the latest version of [TheAgentCompany](https://arxiv.org/abs/2412.14161), revised in September 2025, the best agent completed **30.3%** of them, at an average model cost of **$4.20** per task. The weakest categories were finance, administration and data science, “with many LLMs completing none of the tasks successfully.” Building spreadsheets, collecting information from several people and reading scanned documents are easy for a person and hard for an agent, the authors note. Those are the tasks that fill most mid-market AI lists. Read such a list line by line and most of the items need no agent at all. Routing an invoice to the right approver, pushing a closed deal from the CRM into billing, checking that a new hire’s forms are complete: each is a rule someone can write down. A rule in software runs the same way ten thousand times, costs a fraction of a cent per run, is tested once and can be shown to an auditor. An agent doing the same job costs more per run and may give a different answer the second time. So the most valuable AI decision a COO or CFO can make early is a sort. Every item goes to software, to an agent, or to a person. Then a second filter: an item only gets automated if the work repeats and something other than a person can mark the output wrong. - **30%** Office tasks the best agent finished on its own in [TheAgentCompany](https://arxiv.org/abs/2412.14161), Carnegie Mellon, 2025. - **30–75%** Range of outcome consistency on repeated runs of the same task across 14 models, per [Princeton](https://www.normaltech.ai/p/new-paper-towards-a-science-of-ai), 2026. - **15×** Tokens a multi-agent system uses compared with a chat, per [Anthropic](https://www.anthropic.com/engineering/multi-agent-research-system), 2025. ## Back-office work is where agents score worst The benchmark is useful because it looks like a real workplace. Some admin and HR tasks contain deliberate traps, and one finance task asks the agent to complete a section of IRS Form 6765, the research credit, from company financials on the file server, and to contact the finance director when something is unclear. Coding came out best, partly because models train on public code, the authors suggest, while financial data sits privately inside companies. The models in the paper are a year old, and newer ones score higher. The cost line matters more. At $4.20 per attempt and seven attempts in ten failing, the best agent’s cost per completed task works out to nearly $14. For work a rule could have done, all of it is overhead. ## The same agent gives a different answer the second time The stronger evidence for a CFO is about consistency. In February 2026, Stephan Rabanser, Sayash Kapoor, Arvind Narayanan and colleagues at Princeton published [Towards a Science of AI Agent Reliability](https://arxiv.org/abs/2602.16666). They ran models from OpenAI, Google and Anthropic released over 18 months through two benchmarks, five times per task, with paraphrased instructions. In [their summary](https://www.normaltech.ai/p/new-paper-towards-a-science-of-ai), they write that “nearly two years of rapid capability progress have produced only modest reliability gains.” Many models gave inconsistent answers, “with outcome consistency scores ranging from 30% to 75% across the board,” and performance dropped substantially when the same instruction was reworded. One detail belongs in every finance team’s risk register. On τ-bench, a benchmark where agents serve simulated customers under a company policy, the paper finds that financial accuracy, meaning incorrect charges and refunds, “remains the most common failure mode across all models.” τ-bench itself, published in 2024 by [researchers at Sierra](https://arxiv.org/abs/2406.12045), introduced the measure that matters here: the chance an agent solves the same task correctly every time across eight tries. The best agent then solved about 61% of retail tasks on a single attempt and fewer than **25%** on all eight. The Princeton authors argue that agents need three to five nines, “99.9% to 99.999% accuracy,” before reliability stops being an issue, and “we don’t think LLM-based agents are on track to reach such a threshold.” A tested three-way match rule gives the same answer on every run. ## An agent doing a rules job costs more and proves less Anthropic, which sells agents, says the same in [its guide to building them](https://www.anthropic.com/engineering/building-effective-agents). It recommends “finding the simplest solution possible, and only increasing complexity when needed,” and notes that fixed workflows “offer predictability and consistency for well-defined tasks,” while “agentic systems often trade latency and cost for better task performance.” In its post on its own [research system](https://www.anthropic.com/engineering/multi-agent-research-system), the company reports that agents “typically use about 4× more tokens than chat interactions” and multi-agent systems about 15 times more. The academic version is older and sharper. In [AI Agents That Matter](https://arxiv.org/abs/2407.01502) (2024), Kapoor, Narayanan and colleagues showed that simple baselines beat several elaborate agent designs on a coding benchmark while costing much less, and that for similar accuracy “the cost can differ by almost two orders of magnitude.” [Annelies Gamble](https://x.com/AnneliesGamble/status/2069814714214023186) of Zetta Venture Partners, whose three-way split of work I used in [an earlier note](https://vandam.ai/notes/ai-needs-hr-and-finance), puts the test plainly in the same essay: “If the inputs are structured and the desired behavior can be specified, you probably do not need an agent. You need software.” Her examples are routing an invoice, syncing a CRM field and checking whether required documents are present. > Knowing what not to agentify is the first question to answer when it comes to enterprise deployments – because not every workflow deserves an agent. > > Annelies Gamble, Partner, Zetta Venture Partners, June 2026, [Source](https://x.com/AnneliesGamble/status/2069814714214023186) ## Automate only what repeats and what a machine can mark wrong The second filter comes from software engineering, where agents now run in loops that retry until the work passes. A June 2026 post by [Codez](https://x.com/0xCodez/status/2064374643729773029), an AI content creator summarizing an analysis by the AI newsletter AlphaSignal, lists four conditions for a loop to pay back. Two of them translate directly to operations: “The task repeats. A loop amortizes its setup across many runs.” And: “Verification is automated. The loop needs something that can fail the work without you in the room.” Without an automatic check, every agent run ends with a person reading the output, and the saving turns into a new review job. Anthropic measured that effect on its own staff, which I covered in [the note on its workforce report](https://vandam.ai/notes/inside-anthropic). Finance and operations already own many checks: the bank reconciliation, the match tolerance, the trial balance that must net to zero, the valid GL account list. Where such a check exists, an agent’s output can be failed and retried without anyone in the room. Where none exists, the honest cost of the agent includes the reviewer’s hours. ## Four questions sort every item on the list This is the rule we use. Take each item in this order and stop at the first answer. 1. **Can someone write the rule?** If the inputs come from a system or a form and a new employee could follow the steps from a one-page checklist, it goes to software: a workflow tool, an ERP or CRM rule, a script. Exceptions the rule cannot handle go to a queue. 2. **Who answers for the outcome?** If a mistake would cost a key customer relationship, breach a legal duty, affect someone’s job or move money above an approval threshold, a named person decides. AI can assemble the file, and the person signs. The same applies when nobody can yet describe what a good result looks like. 3. **Does it repeat?** My rule is at least weekly and at least a few dozen times a month. Below that, the setup, testing and upkeep of an agent will not pay back, and a person with a good chat assistant is the cheaper answer. 4. **Can a machine mark it wrong?** If an existing control, a reference total or a second system can fail the output, it is an agent candidate. If only a person can judge it, either budget the review hours into the business case or leave it manual for now. ## Ten typical requests, sorted A list of the kind a 400-person distributor might hand its AI team, run through the four questions: 1. **Route vendor invoices to the right approver.** Software. Cost center and amount decide the approver; the ERP’s approval workflow already does this. 2. **Match invoices to purchase orders and receipts.** Software. A tolerance rule clears the bulk automatically. 3. **Push closed-won deals from the CRM into billing.** Software. A field mapping and an integration, with failed records sent to a queue. 4. **Check that every new hire’s onboarding forms are complete.** Software. A checklist in the HR system with reminders. 5. **Build the weekly cash report from bank balances and receivables aging.** Software. A scheduled report; both sources are structured. 6. **Answer customer emails asking where an order is.** Agent. The wording varies and the agent has to find the order, but the answer can be checked against the shipping system before it goes out. 7. **Resolve the invoice match exceptions.** Agent. Gathering the PO history, receiving notes and supplier emails takes judgment, and the proposed resolution must again pass the tolerance check before a clerk approves the posting. 8. **Compare inbound supplier contracts with the standard terms.** Agent, with counsel signing off. Formats vary, the clause library is the check, and a lawyer decides on any deviation it flags. 9. **Set the credit limit for a large new customer.** Person. An agent can assemble the financials and payment history; the controller or CFO signs, because the exposure is theirs. 10. **Write each employee’s performance review.** Person. The trust between a manager and a report is the work, and no machine can mark the review wrong. Five of ten go to software, three to an agent, two to a person. The proportions will differ by company, but a large share of back-office requests are rules nobody has automated yet. Check whether some of those five are features the company already pays for in its ERP or HR system. ## Every agent still needs someone who can build and fix it The sort also changes who you need. Visual agent builders are sold as tools for operations staff. [Noah Epstein](https://x.com/NoahEpstein_/status/1978436984591745202), who promotes a competing tool in the same post, wrote after testing the main ones in October 2025: “They’re ALL ‘no-code for people who can code.’” His list of what each still requires reads like a job description: API authentication, webhook logic, error handling, data transformation and conditional branching. Treat it as one builder’s claim, made while selling an alternative. The software bucket needs a systems analyst who knows the ERP. The agent bucket needs that person plus an owner for the checks and test cases who reads the failures every week. Whatever ends up in an agent also enters your control environment, which I described in [the note on agents and the ledger](https://vandam.ai/notes/agents-and-internal-controls). A CFO should budget for those people as a line in the business case, next to the licenses. ## What to do with your list this month Put the current AI list in one spreadsheet with four columns: the rule in one sentence (or “cannot write it”), the person who answers for the outcome, the monthly volume, and the check that can fail the output. Fill it in a 90-minute session with the process owners; most items sort themselves once someone tries to write the rule. Fund the software rows first. They are the cheapest, they can be tested and audited, and they leave cleaner data for the agents that come later. For every agent row, the CFO should ask for three things before approving it: why a rule cannot do the job, which existing check will mark its output wrong, and the cost per completed task including review hours, the measure I set out in [the note on pricing AI per finished task](https://vandam.ai/notes/cost-of-the-i). Rows with a named owner and no check stay with that owner. ## More notes 29 October 2026 6 min read ## Who owns what your AI learns. Y Combinator now asks for startups that help companies own their intelligence. Before a mid-sized company owns a model, it can own the four things every model works with: context, evals, rules and connections. 26 October 2026 9 min read ## Delete the step first and give the agent whatever survives. Musk’s five-step method puts automation last, and most AI programs start there because an agent is the one step a vendor can sell. I walk accounts payable through the steps in his order and show why an agent that clears exceptions fast hides the steps that should go. ## Your agents need a critic and so does the plan they carry out. Models grade their own work too kindly, and the labs now separate the agent that makes from the agent that checks, the way finance separates maker and checker. I set out a four-layer review design with a minutes budget, and why the brief itself needs a critic with no stake in it.