Agent Evaluation Dataset Design: Test What Can Fail

Agent evaluation dataset design starts with a question: what would make this agent appear competent without doing the work? A dataset earns its place by ruling out those shortcuts. Include tasks that require the intended evidence, controls that change the correct outcome, and verifiers that inspect effects rather than accepting a confident explanation.

The takeaway is simple: design the task, its counterexample, and its evidence contract together. This is a design guide, not an experiment report. The examples below are proposed synthetic fixtures; they do not represent measured model performance.

Define the decision an agent evaluation must support

A release gate, a capability study, and an estimate of production reliability need different datasets.

A release gate protects known behavior. A capability study probes work the current system cannot reliably complete. A production estimate needs a sampling procedure that reflects the population you intend to describe. Combining these into one undifferentiated score makes an improvement difficult to interpret.

Anthropic’s guide to agent evaluations distinguishes capability evaluations from regression evaluations and separates a task, a trial, a grader, and an environmental outcome. That vocabulary prevents a common counting mistake: repeated attempts at one task are not additional independent tasks.

Write the decision before collecting examples: “Block changes that introduce unauthorized writes,” or “Compare two retrieval policies on evidence-dependent support tasks.” Specify what remains outside scope. A read-only question-answering suite cannot establish safe write behavior, regardless of its answer accuracy.

The broader task-level evaluation framework starts with user outcomes. Dataset design makes those outcomes concrete enough to challenge and reproduce.

Sample task families, not just prompt wording

Begin with a coverage map of the work. For a support agent, useful families might include factual lookup, policy application, conflicting records, missing authority, and recovery after an uncertain write.

Within each family, record the relevant variation: evidence location, number of required sources, policy precedence, authorized actions, and environmental failure conditions. Ten paraphrases of one lookup are less diverse than ten tasks that exercise different decisions.

Use production-derived examples only after removing sensitive data and preserving the structure that caused the difficulty. Replacing every entity with the same placeholder can destroy a record-matching problem. Removing a timestamp can erase the reason one source superseded another. Sanitization needs an answerability review, not merely a redaction pass.

Synthetic fixtures are useful for controlled variation, but label them and review them. A generator can accidentally encode the answer in record names, output length, or the order of options. A collection created by one template may measure recognition of that template rather than the operational skill you care about.

OpenAI’s evaluation guidance recommends task-specific evaluations, a mixture of data sources, and typical, edge, and adversarial cases. Treat these as separately identified slices. If you deliberately oversample rare failures, report that choice rather than presenting the raw average as the expected production success rate.

Make task difficulty depend on the intended skill

Longer is not necessarily harder. An agent may answer a question from its prior knowledge while ignoring a large attached document. Adding irrelevant text then measures the cost of processing text, but may not test whether the agent can recover a necessary fact from it.

Construct difficulty around a dependency. A task might require joining two records, applying an explicit exception, choosing the current revision, or declining an action when authorization is missing. The expected outcome should change when that decisive evidence changes.

Keep the first version small enough to inspect manually. Confirm that the authorized inputs actually determine an acceptable result and that the tools can reach the needed evidence. An impossible fixture is not an advanced reasoning test unless recognizing impossibility is the stated objective.

Separate structural difficulty from observed difficulty. “Requires two-source reconciliation” describes a fixture. “The baseline rarely succeeds” describes measured behavior under a particular configuration. Do not assign the second label without runs.

Use a readiness pilot to find ambiguous instructions, broken environments, and overly narrow graders. Keep those pilot attempts separate from the final comparison. After inspecting pilot failures and revising tasks, freeze a new dataset version; do not quietly mix the earlier attempts into the primary result.

Use negative controls to expose evaluation shortcuts

For an illustrative policy task, suppose a synthetic record says a customer’s eligibility window is fourteen days. The agent must determine eligibility from that record and the supplied purchase date. A paired version changes the policy window while preserving the surrounding format and adjusting every dependent reference consistently.

If the answer stays unchanged when the correct decision must change, the system may be ignoring evidence. If it changes when only an irrelevant identifier changes, it may be relying on a shortcut. Neither observation alone establishes the internal cause; traces and additional controls help distinguish retrieval, reasoning, and grading failures.

Build a small control matrix before increasing dataset size:

Control	What changes	What the check asks
Changed decisive fact	A policy value or authoritative record	Does the answer follow the new evidence?
Removed evidence	The only source needed for the decision	Does the agent report uncertainty or use the defined escalation path?
Irrelevant perturbation	Unused identifiers or distractor wording	Does the valid outcome remain stable?
Authority contrast	The same requested effect with and without valid authority	Does execution respect the permission boundary?
Benign counterpart	Hostile instructions replaced by ordinary task content	Can the agent still complete legitimate work?

These are proposed checks, not reported passes. Evidence-removal tasks require a defined fallback: if another authorized tool can retrieve the missing fact, retrieval may be the correct behavior rather than abstention.

For adversarial tasks, follow the same distinction used in red-teaming agent guardrails: avoiding forbidden effects and completing useful work are different outcomes. An agent that refuses everything can look excellent on a safety-only score and fail the product’s purpose.

Prevent dataset contamination across the whole harness

Holdout isolation is broader than excluding rows from model training. Test answers can leak through prompt examples, retrieval indexes, persistent memory, shared workspaces, tool caches, or previous evaluation feedback.

Split at the level of the underlying case or task family where appropriate. Variants of the same incident, repository issue, document, or customer scenario should not cross development and holdout boundaries unnoticed. Randomly splitting near-duplicate prompts creates a weak test of generalization.

Store answer keys and grader-only metadata outside the agent’s accessible environment. A hidden answer column is not hidden if a filesystem tool can read the dataset. Reset writable state between independent trials, and define which caches or memories are intentionally retained. A persistent-memory evaluation needs its own lifecycle contract; accidental carryover is not that contract.

Keep an exposure ledger: which tasks developers examined, which traces informed prompt changes, and when a holdout was consulted. Once repeated holdout feedback guides optimization, describe that set as a reused evaluation set and reserve fresh cases for the next independent check.

For public benchmarks, record known exposure limitations. A private holdout can reduce application-level leakage but does not prove that a foundation model has never encountered similar material. Avoid claiming a dataset is contamination-free without evidence that supports that scope.

Give every evaluation case an evidence contract

A task record needs more than a prompt and an ideal answer. Retain its identifier, family, provenance, fixture version, trusted instructions, available tools, authority limits, acceptable outcomes, prohibited effects, and grader version.

Keep exact input artifacts or content-addressed references to them. Preserve the initial environment, observable tool requests and results, final output, terminal status, and independently inspected final state. Record model and harness configuration, trial identifiers, timing, and usage when available. Do not promise access to hidden model reasoning; evaluate the observable behavior and resulting artifacts.

Grade the result without requiring one preferred trajectory unless the path itself is a requirement. A correct, authorized solution should not fail merely because it used a different valid lookup order. Conversely, a correct answer should not excuse an unauthorized side effect.

Review graders with deliberately wrong and alternative-valid examples. Check that they reject unsupported conclusions, accept permitted variations, and detect effects the agent denies taking. If a model judges open-ended quality, version its rubric and calibrate a sample against human judgments. Keep unresolved disagreements visible.

Missing evidence is its own status. A crashed runner, disconnected sink, or absent final-state snapshot cannot establish successful completion. Define exclusions before running, retain every attempt’s status, and report both valid-trial results and infrastructure failures. Otherwise, dropping inconvenient runs can quietly improve the score.

Report coverage and uncertainty before the headline score

Publish the number of distinct tasks, families, trials per task, and outcomes by slice. Repeated trials estimate variability on those tasks; they do not create broader task coverage. Paired controls also share structure, so avoid treating every row as independent evidence of generalization.

For a harness comparison, use the same frozen cases, reset procedure, budgets, and grading rules. Where execution order could matter, predefine an interleaved or randomized schedule and record it. Distinguish first-attempt success from success after retries, and include the full permitted retry cost.

If reporting an uncertainty interval, explain the sampling unit and method. Task-level or family-level resampling may be more appropriate than treating correlated trials as independent. A small convenience suite can identify regressions without supporting a precise claim about all production traffic.

Finally, route discovered failures into behavior-verified regression gates. Keep the useful cases, but do not erase their history: a once-unseen holdout case that now drives development serves a different role.

A trustworthy dataset is not the largest folder of prompts. It is a versioned argument that the agent needed the right evidence, respected the right boundaries, and produced an outcome someone else can verify.