Eval Suite by VANDAM: Test every new model on your own work

This page as Markdown llms.txt llms-full.txt services.json sitemap.xml
# Eval Suite by VANDAM: Test every new model on your own work

> A test suite built from your own work, a safe place to run it and one command that compares any model with the one in production. You see quality, cost and speed per case, and the switch becomes a decision with evidence behind it.

URL: https://vandam.ai/eval-suite  
Contact: hendrik@vibelab.be

# Know whether a new model helps before you switch.

A test suite built from your own work, a safe place to run it and one command that compares any model with the one in production. You see quality, cost and speed per case, and the switch becomes a decision with evidence behind it.

[Discuss the project](mailto:hendrik@vibelab.be?subject=VANDAM%20/%20Eval%20Suite)

**$12,800** three weeks

## Most teams can watch their agents. Far fewer test them.

- **32%**
  of 1,340 practitioners named quality as the top barrier to putting agents into production. [LangChain](https://www.langchain.com/state-of-agent-engineering), State of Agent Engineering.
- **52%**
  run offline evaluations on test sets, while 89% have some form of observability for their agents. [LangChain](https://www.langchain.com/state-of-agent-engineering), same survey.
- **62%**
  have tracing detailed enough to inspect individual agent steps and tool calls. [LangChain](https://www.langchain.com/state-of-agent-engineering), same survey.

> At dinner, someone asked the table, “If a new model drops next month, does your company get better or worse?” HALF the table went quiet!
>
> Greg Isenberg, CEO, Late Checkout, October 2026, [Source](https://x.com/gregisenberg/status/2110414696037716372)

So that dinner question gets an answer the same afternoon.

## Few companies can say how accurate their AI is. A new model every month makes that harder.

An eval is a test for an agent: a real task, the answer your people would accept and a check that says right or wrong. Run them together and you know how well the agent does the job today, and whether a change makes it better or worse. Without them, every model release becomes a debate settled by a few tries in a chat window.

[Alex Lieberman](https://x.com/businessbarista/status/2097478065723162959) summed up the recipe after a session on evals with LangChain: tasks, a verifier and an environment that looks real to the agent but cannot overwrite production. Each time a new model comes out, you run the suite and weigh cost, speed and intelligence against each other. In his list of thirty features of an AI-native company, [evals are core infrastructure](https://x.com/businessbarista/status/2094213970215231831).

We build that suite for one to three of your agents. The tasks come from your own records. The rubrics are written with the people who do the work, because they decide what counts as right. Tracing is on, so a failed case shows the step where it went wrong. The suite never runs against live systems, and it stays with you: cases, rubrics and the runner sit in your repository, with the rest of [what stays yours](https://vandam.ai/models). The expert decisions in a [Judgment Vault](https://vandam.ai/judgment-vault) become cases here, and the reasoning is in [own your intelligence](https://vandam.ai/notes/own-your-intelligence).

> In the world of agents, performance management is evals.
>
> Garrett Lord, CEO, Handshake, June 2026, [Source](https://x.com/GarrettLord/status/2068754262440767500)

## How the suite gets built.

1. Take the cases from the work: We pull real tasks from your records for each agent: requests, files, tickets and the answers your people sent. Edge cases go in on purpose, such as the order with no quantity or the discount just over the limit.
2. Write down what right means: Rubrics and verifiers are written with the people who do the work. Some checks are code, such as the price on the contract list. Others are a checklist that a second model or a person applies. Each check is one plain line.
3. Run it the same way every time: The suite runs in a test environment with copies of your systems and tracing switched on. One command runs any model against it and writes a report with quality, cost per task and speed side by side, case by case.

## Built once. Run on every model that matters.

Models ship every few weeks and your work changes with them. After the handover we can run the suite on each release worth testing, turn production failures into new cases and tell you when a switch pays off.

- New model runs: Each relevant release run against your suite within a week, with the report.
- New cases: Failures from production turned into cases with the expected answer attached.
- Cost and quality: Cost per task and pass rate per agent, tracked month by month.
- Review: A monthly session with the rubric owners on what changed and what to switch.

[Ask about model watch](mailto:hendrik@vibelab.be?subject=VANDAM%20/%20Eval%20Suite%20care)

## What you receive.

- Task set: Real cases with expected answers, for one to three agents.
- Rubrics and verifiers: The checks that decide right or wrong, one plain line each.
- Model comparison: Quality, cost and speed per model, case by case.

$12,800

three weeks

An indication from past projects. The final price can move a little with the size and complexity of your company, and we agree it in writing before we start.

For operations and technology leads with one to three agents, live or planned. Useful when a new model ships and nobody can say whether to switch.

[hendrik@vibelab.be](mailto:hendrik@vibelab.be?subject=VANDAM%20/%20Eval%20Suite)

## Where the suite leads. Each next step has its own service.

### Agent Ops

We run the suite every month on the live agents, and before every model switch.

**$3,200/mo**

### Token Ledger

The cost side of every verdict: what the switch saves per task.

**$9,600 + $1,600/mo**

### Private AI

Tests open models you could run yourself against the ones you pay for.

**$16,000**

## Before you ask.

### How many cases does a suite need?

Enough to cover the work and its edge cases. For a focused agent that is often 40 to 200. We start with the cases that matter most and grow the set from production.

### We have no agent live yet. Is this too early?

A suite written before the build tells the builders what right means, and it lets you compare models before you choose one.

### Which models can you compare?

Any model you can reach through an API or run yourself, including open-weight models on your own hardware. We usually start with Claude and test the alternatives against it.

### Does this touch our live systems?

No. The suite runs in a test environment with copies or simulations of your systems. Agents optimize for the score they are given, so they never practice on production.

### Who owns the suite?

You do. Cases, rubrics, verifiers and the runner live in your repository and run without us.