Evals for Agentic Systems: Testing the Harness, Not Just the Model
Model benchmarks don't predict agent performance. How to build task-level evals: golden traces, scenario suites, regression gates, scoring harness design.
A model benchmark scores a single call. An agent runs hundreds of them through a loop you wrote, against tools you built, on context you selected. The benchmark cannot see any of that, which is why it does not predict what your users experience.
Task-level, not token-level
Score the outcome the user cares about, then keep the traces that explain it. An eval you cannot debug is a number, not an instrument.
01 · Golden traces
Keep the whole run
Store the full call sequence for passing runs, not just the score. Regressions are usually visible as a changed shape before they show up as a changed number.
02 · Scenario suites
Fixed, small, adversarial
A suite you can run on every change beats a large one you run quarterly. Add the failure you just shipped as a case.
03 · Regression gates
Block on the metric that matters
Gate merges on task completion and cost, and let everything else be a dashboard.
"Model benchmarks do not predict agent performance. Eval the harness."
The experiment: the same task suite scored by three harness configurations, with the scoring code and every trace published alongside the result. Those traces are also the connective tissue between evals and observability — production failure traces become eval cases, eval patterns become guardrails. The same suite is what makes a model swap drill possible: run both models against it, diff the traces, gate the switch. Ground-truth eval cases are level 4 on the verifier spectrum — the strongest automatic check, and the narrowest.