VANDAMAgent Ops
Keep your agents workinglong after go-live.
A monthly retainer to run your live agents. We read their traces, turn every failure into a fix and a test, re-run the eval suite whenever a model or your data changes, and send a KPI report each month and a shortlist of what to automate next each quarter.
$3,200/mo monthly · six-month minimum
October 31
Month closed
- runs
- 1,534
- completed without help
- 99.9%
- failed
- 1
- hours saved
- 224
Invoice agent · run 14-0912 · Brightline Packaging, BP-20981.pdf
{ "trigger": "mailbox", "from": "billing@brightline.example", "attachments": 1 }{ "pages": 1, "lines": 9, "total": -412.00, "header": "Brightline Packaging · BP-20981", "footer": "Credit note · ref BP-20417" }{ "po": "4502", "supplier": "Brightline Packaging", "status": "open", "lines": 9 }{ "input": "header", "label": "invoice", "confidence": 0.94, "footer_read": false }{ "matched": 9, "price_diff": 0, "qty_diff": 0 }{ "posted": false, "check": "amount_sign", "document_total": -412.00, "posting_total": 412.00 }{ "status": "failed", "incident": "INC-0412", "owner": "Agent Ops" }
Fix written, 1 h 48 min after the failure
Read the document type from the header and the footer. A negative total never posts as an invoice; it goes to the review queue.
148 of 148 pass · eval suite, with the failed run added as case 148
October closed: 1,534 runs by 4 agents, 99.9% completed without help, 1 failure, fixed in 1 h 48 min and added to the suite, about 224 hours saved.
Most teams run agents in production.
About half test them.
- 57%
- of organizations have agents running in production. LangChain, State of Agent Engineering, survey of 1,340 professionals.
- 52%
- run offline evaluations on test sets, and 37% run online evals on real-world performance. LangChain, same survey.
- 32%
- name quality as a top barrier to production, the most cited barrier in the survey. LangChain.
Turn on tracing first. Traces = receipts of every action (tool calls, Salesforce pings, web searches, dead ends).
TenexSeptember 2026Source The trace shows where it went wrong, usually a few steps before the error.
Agents drift after launch.
Someone has to own them.
An agent that worked in its first week meets new cases every week after. A supplier changes its invoice layout, a model version is retired, a field in the CRM gets renamed. The agent keeps running while some of its results quietly get worse, and without someone reading the traces, the first sign is a customer or a controller finding the mistake.
Agent Ops is that someone. We review the traces of every run, the record of each step and tool call, and catch failures, loops and slow drift. Each failure becomes a fix and a new case in the eval suite, so the same mistake cannot come back unnoticed. When a model or a data source changes, the whole suite runs again before the change reaches your team.
“Who owns a build after it’s deployed?” is on the list of questions Alex Lieberman’s team hears most from enterprise executives. Many companies will hire for it. 
Aaron Levie describes the role as an agent deployer and manager who “manages evals and reviews after any major model or data change, and runs and manages the agents on an ongoing basis tracking KPIs.” Until that person is in place, you can rent the job by the month, and hand it over when they start.
The more enterprises I talk to about AI agent transformation, the more it’s clear that there is going to be a new type of role in most enterprises going forward. The job is to be the agent deployer and manager in teams.
BoxApril 2026Source What your team sees every month.
The run log opens on the month: every agent, every run, with failures marked and each one linked to its trace, its fix and the eval cases it added. Each agent shows its completion rate, its cost and the hours it saved.
Fixes and eval results are written down next to the failures they answer, and a model update only goes live with its suite result in the log. Anyone can ask what went wrong last week and get the trace, the fix and the test.
What every month includes.
Your agents and your stack differ. The month runs the same way.
Tracing
Every run recorded step by step, in your environment.
Daily review
Failures, loops, slow runs and cost spikes flagged.
Root cause
Each failure traced to a prompt, tool, data or model change.
Fix
Written, tested and shipped through your release process.
Eval suite
Failing cases added, the suite re-run before every change.
Change watch
Model updates and data changes tested before users see them.
KPI report
Runs, completion, time to fix and hours saved, per agent.
Next to automate
A ranked shortlist each quarter, with the expected gain.
How a month of Agent Ops runs.
Read the traces
Tracing is switched on for every agent we run, in your own observability tool or in one we set up in your environment. We review runs every working day, flag failures, loops and cost spikes, and keep a log of what we found.
Fix and test
Each failure is traced to its cause: a prompt, a tool, a data change or a model change. We write the fix, add the failing case to the eval suite and re-run the suite before the fix goes live through your release process.
Report and look ahead
At the end of each month you receive the KPI report per agent: runs, completion rate, failures, time to fix and hours saved. Each quarter we bring a ranked shortlist of what to automate next, with the expected gain.
Run by us.
Yours to take in-house.
The six-month minimum gives the eval suite and the runbook time to cover the cases your agents actually meet. Everything lives in your environment: traces, evals, fixes and the runbook. When you hire your own agent manager, we hand it over and train them.
Trace review
Runs read every working day, failures logged with their trace.
Eval re-runs
The full suite run after every model, prompt or data change.
KPI report
Monthly, per agent, next to the month before.
Handover
Runbook, evals and training for the person who takes over in-house.
What you
receive.
Trace review
Every agent run read for failures, loops and quiet errors.
Failure-to-fix loop
Each failure traced, fixed and added to the eval suite.
Eval re-runs
The full suite re-run after any model, prompt or data change.
KPI report
Runs, completion rate, time to fix and hours saved, per agent.
Where Agent Ops leads.
Each next step has its own service.
Right now nobody knows which of their agents are actually good and which ones are quietly wasting money.
Late CheckoutApril 2026Source Before you ask.
Which agents can you run?
Agents we built and agents built by your team or another partner, as long as tracing can be switched on and the agent can run against an eval suite. If there is no suite yet, building one is part of the first month.
Do you need access to production?
Read access to traces and logs, and a test environment for the evals. Fixes go live through your own release process, with your sign-off.
What counts as a failure?
A run that ended in an error, produced a wrong result, looped or cost far more than usual. We agree the thresholds per agent in the first month.
What happens when a model is updated?
The eval suite runs on the new version before your agents switch. You see the result next to the current version and decide.