VANDAMEval Suite
Know whether a new model helpsbefore you switch.
A test suite built from your own work, a safe place to run it and one command that compares any model with the one in production. You see quality, cost and speed per case, and the switch becomes a decision with evidence behind it.
$12,800 three weeks
Model B came out this morning.
Before anything changes, Atlas & Co. runs the 40 quote requests its sales team wrote down, against the model in production and the new one.
Model A, in productionModel B, released this morning
Case 07 · Meridian Clinics
“We’d like 18% off 120 boxes of gloves.”
- Expected
- 18% held for Finance approval
- Model A
- 18% applied without approvalfails
- Model B
- 18% held for Finance approvalpasses
Decided byA discount above 12% names Finance as the approver.
Cases passed
Cost per quote
Median time per quote
Switch the quotes agent to Model B. It passes 37 of 40, fixes 5 cases Model A got wrong and breaks 2, at 26% less per quote.
Most teams can watch their agents.
Far fewer test them.
- 32%
- of 1,340 practitioners named quality as the top barrier to putting agents into production. LangChain, State of Agent Engineering.
- 52%
- run offline evaluations on test sets, while 89% have some form of observability for their agents. LangChain, same survey.
- 62%
- have tracing detailed enough to inspect individual agent steps and tool calls. LangChain, same survey.
At dinner, someone asked the table, “If a new model drops next month, does your company get better or worse?” HALF the table went quiet!
Late CheckoutOctober 2026Source So that dinner question gets an answer the same afternoon.
Few companies can say how accurate their AI is.
A new model every month makes that harder.
An eval is a test for an agent: a real task, the answer your people would accept and a check that says right or wrong. Run them together and you know how well the agent does the job today, and whether a change makes it better or worse. Without them, every model release becomes a debate settled by a few tries in a chat window.
Alex Lieberman summed up the recipe after a session on evals with LangChain: tasks, a verifier and an environment that looks real to the agent but cannot overwrite production. Each time a new model comes out, you run the suite and weigh cost, speed and intelligence against each other. In his list of thirty features of an AI-native company, 
evals are core infrastructure.
We build that suite for one to three of your agents. The tasks come from your own records. The rubrics are written with the people who do the work, because they decide what counts as right. Tracing is on, so a failed case shows the step where it went wrong. The suite never runs against live systems, and it stays with you: cases, rubrics and the runner sit in your repository, with the rest of what stays yours. The expert decisions in a Judgment Vault become cases here, and the reasoning is in own your intelligence.
In the world of agents, performance management is evals.
HandshakeJune 2026Source What the team opens when a new model ships.
The report opens on the verdict: switch, stay or test further, with pass rate, cost per task and median time for each model next to the one in production. Under it sit the cases that changed, first the ones the new model fixed, then the ones it broke, each with the rubric line that decided it.
The people who wrote the rubrics keep them current. When an agent gets something wrong in production, the trace becomes a new case with the expected answer attached, and the next run checks it. Every past run stays in the history, so you can see how each model and each change moved the score.
What every eval suite contains.
Your agents differ. The suite is built the same way each time.
Task set
Real tasks from your records, for one to three agents.
Expected answers
What your people would accept for each task, agreed with them.
Edge cases
The hard and odd cases, put in on purpose.
Rubrics
What right means, one plain line per check.
Verifiers
Code, a second model or a checklist that applies each rubric line.
Test environment
Copies or simulations of your systems, apart from production.
Tracing
Every step and tool call recorded, so a failure points to its cause.
One-command runner
Any model against the suite, with quality, cost and speed side by side.
How the suite gets built.
Take the cases from the work
We pull real tasks from your records for each agent: requests, files, tickets and the answers your people sent. Edge cases go in on purpose, such as the order with no quantity or the discount just over the limit.
Write down what right means
Rubrics and verifiers are written with the people who do the work. Some checks are code, such as the price on the contract list. Others are a checklist that a second model or a person applies. Each check is one plain line.
Run it the same way every time
The suite runs in a test environment with copies of your systems and tracing switched on. One command runs any model against it and writes a report with quality, cost per task and speed side by side, case by case.
Built once.
Run on every model that matters.
Models ship every few weeks and your work changes with them. After the handover we can run the suite on each release worth testing, turn production failures into new cases and tell you when a switch pays off.
New model runs
Each relevant release run against your suite within a week, with the report.
New cases
Failures from production turned into cases with the expected answer attached.
Cost and quality
Cost per task and pass rate per agent, tracked month by month.
Review
A monthly session with the rubric owners on what changed and what to switch.
What you
receive.
Task set
Real cases with expected answers, for one to three agents.
Rubrics and verifiers
The checks that decide right or wrong, one plain line each.
Model comparison
Quality, cost and speed per model, case by case.
Where the suite leads.
Each next step has its own service.
Before you ask.
How many cases does a suite need?
Enough to cover the work and its edge cases. For a focused agent that is often 40 to 200. We start with the cases that matter most and grow the set from production.
We have no agent live yet. Is this too early?
A suite written before the build tells the builders what right means, and it lets you compare models before you choose one.
Which models can you compare?
Any model you can reach through an API or run yourself, including open-weight models on your own hardware. We usually start with 
Claude and test the alternatives against it.
Does this touch our live systems?
No. The suite runs in a test environment with copies or simulations of your systems. Agents optimize for the score they are given, so they never practice on production.
Who owns the suite?
You do. Cases, rubrics, verifiers and the runner live in your repository and run without us.