Agent Evaluation Uncertainty: When Not to Ship

Agent evaluation uncertainty is the gap between an observed score and the conclusion you want to draw from it. A candidate can finish ahead on a dashboard while the evidence still permits a meaningful regression. More trials do not automatically close that gap if they repeat the same narrow collection of tasks.

The takeaway: define the population, preserve the independent unit, and let a release comparison remain inconclusive. An uncertainty interval is useful only when its assumptions match the evaluation design. This is a design guide, not an experiment report; the decision rules below are proposed policies, not measured results.

Define what an agent evaluation score estimates

Start with the quantity you intend to estimate. Is it the average success probability across a fixed regression suite? Performance on new tasks sampled from a specified workload? Reliability over repeated attempts at one difficult task? These questions need different evidence.

For a fixed suite, the task list is the object of interest. Additional trials can reduce uncertainty about behavior on those tasks. They do not establish performance on unseen workflows. For a workload estimate, task selection contributes uncertainty too: another representative sample could contain a different mix of difficulties.

Write down the unit of work, target population, sampling procedure, task weights, trial budget, success definition, and time period. A deliberately adversarial collection can reveal failure modes without estimating their production frequency. A convenient set of support tickets does not become representative merely because every ticket has a trace.

The agent evaluation dataset-design guide covers this coverage problem. Statistical precision cannot compensate for omitted task families, contaminated examples, or an evaluation environment that removes the hardest operational constraints.

Repeated trials are not independent tasks

Anthropic’s agent-evaluation guide distinguishes a task from a trial: a task defines the problem and success criteria; a trial is one attempt. Retain both identifiers in the results.

Repeated trials tell you about run-to-run behavior for a particular task under the tested configuration. They do not create new independent examples of task difficulty. If several tasks are paraphrases of one template, even the task count can overstate independent coverage.

Consider three levels of variation:

Level	What changes	What more observations can clarify
Trial	Stochastic execution of the same task	Reliability conditional on that task and environment
Task	Inputs, required reasoning, or operational case	Variation across cases in the sampled workload
Shared cluster	Customer, repository, template, or incident	Dependence among related cases

The appropriate independent unit may be a repository rather than a coding issue, or an incident rather than each support message it produced. Choose it from the data-generating process, not from the row format that makes the interval smallest.

Trials can also share infrastructure failures or time-dependent provider behavior. Reset mutable state, isolate side effects, and record execution blocks. Merely labeling random seeds does not make calls independent. Report unique tasks, higher-level clusters, scheduled trials, completed trials, and unresolved evaluations separately.

When budgeting, ask which uncertainty matters. More repeats help identify flaky behavior on existing tasks. More genuinely independent tasks help characterize workload variation. A credible broad-performance claim usually needs the latter, even when repeating familiar tasks is cheaper.

Separate sampling uncertainty from grader error

A confidence interval around incorrect labels is still about incorrect labels. If the grader accepts fabricated completion, repeating the evaluation can produce a precise estimate of the wrong outcome.

Freeze the scoring contract for a release comparison. Audit false passes, legitimate alternatives, and missing evidence using the methods in agent evaluation graders. Then describe what the statistical analysis treats as observed: deterministic outcome checks, adjudicated judgments, or an explicitly modeled noisy measurement process.

Do not fold grader errors into stochastic agent failures without explanation. A missing transaction record is an unresolved evaluation, not proof that the transaction failed. Conversely, a genuine agent timeout under the declared product deadline may be a task failure even if the evaluator itself functioned correctly.

Predefine how each category affects the denominator and release gate. Report unresolved coverage by arm and task family. Quietly discarding cases that one candidate makes hard to grade can bias the comparison. If required evidence is missing, the appropriate release result may be inconclusive regardless of the interval computed on the remaining cases.

Use paired comparisons for candidate releases

Evaluate baseline and candidate on the same task cases with equivalent initial state, permissions, budgets, and graders. For each task, estimate each arm’s success rate from its scheduled trials. Subtract baseline from candidate, then aggregate those task-level differences using the declared weights.

With equal task weights, the point estimate is the mean of those differences. If trial counts differ, pooling all successful trials and dividing by all trials changes the weighting: heavily repeated tasks gain influence. Use that pooled quantity only if it is the metric you deliberately chose.

Pairing removes task-mix differences from the comparison. It does not require pairing the first stochastic completion from one arm with the first from the other. Trial-level pairing needs a real shared experimental unit, such as a controlled scenario seed or matched environment block. Preserve that structure only when it exists.

Interleave or randomize arm execution within planned blocks to reduce confounding from service changes. Restore clean state between attempts. An agent that inherits records written by its predecessor is not receiving the same task.

Inspect the distribution of task-level changes, not only the mean. A candidate can improve common lookups while regressing on authorization checks. Report predeclared slices and retain per-task traces. An overall improvement cannot waive a mandatory invariant.

Match confidence intervals to the sampling unit

For a workload comparison with enough independently sampled task cases, a paired task-level bootstrap is one possible analysis. Resample task identifiers with replacement; carry each task’s baseline and candidate records together; recompute the weighted comparison according to the declared sampling design.

SciPy’s bootstrap documentation describes resampling and a paired option that uses the same sampled indices across inputs. That option preserves row pairing. It does not discover which rows belong to the same customer, template, or repository. Your analysis must construct the correct units before calling a library.

If related tasks share a cluster, resample independent clusters rather than pretending their member tasks are unrelated. If the design samples within strata, preserve that design and its target weights. A naive bootstrap over all rows can answer a different question from the one your sampling plan intended.

For a fixed-suite claim, resampling tasks instead introduces a hypothetical changing task mix. Analyze trial variability conditional on the fixed tasks instead. If both task sampling and stochastic execution matter, choose a method that represents those levels; do not automatically add an inner trial bootstrap to an outer task bootstrap. Depending on the model, that can count variability already present in the observed task estimates twice.

Document the interval method, confidence level, resampling unit, weights, and assumptions. Bootstrap intervals are approximate, not a universal repair for small samples. Very few independent clusters, rare failures, or identical observed outcomes can make ordinary bootstrap estimates unstable or degenerate. Use an appropriate design-specific method or report that the available data cannot support the intended claim.

A confidence procedure’s coverage describes its behavior over repeated sampling under its assumptions. It is not the probability that a particular observed interval contains the true effect, and it does not cover distribution shift or unmeasured grader bias.

Make inconclusive a real release decision

Choose the release rule before seeing candidate scores. Separate a claim of improvement from a claim of acceptable non-inferiority. The latter permits a bounded loss; it does not establish a gain.

For an illustrative non-inferiority policy, let the effect be candidate success minus baseline success. Let the tolerated loss be a positive margin chosen from product consequences, not from whichever margin the candidate happens to clear. Define the confidence-bound procedure in advance, including whether it is one-sided or two-sided.

Evidence under the declared procedure	Quality-gate interpretation
Lower bound is above the negative tolerated-loss margin	Supports non-inferiority, subject to other gates
Upper bound is below the negative tolerated-loss margin	Supports an unacceptable regression
Interval crosses the negative tolerated-loss margin	Inconclusive for non-inferiority
Required outcome evidence or mandatory checks are missing	No clean release verdict

Boundary equality should follow the written policy; a strict rule does not accept equality as passing. To claim improvement, use a separately specified rule requiring evidence of a positive effect. Passing a non-inferiority gate is not evidence that the candidate is better.

“Inconclusive” needs an operational action. Keep the baseline, obtain the missing independent evidence, repair the evaluator, or narrow the deployment scope under a separately authorized policy. Do not translate “we cannot establish a regression” into “we established equivalence.” The agent regression-gate framework provides the surrounding machinery for mandatory checks, evidence, and release ownership.

Stop when the protocol says to stop

Repeatedly checking a conventional fixed-sample interval and stopping at the first favorable result can invalidate its advertised coverage. So can selecting the best of many candidates on the same held-out suite and presenting its score as though no selection occurred.

Choose the evaluation budget and stopping rule in advance, or use a sequential procedure designed for repeated inspection. Record candidate selection and keep an untouched confirmation set where the decision requires one. Predeclare important slices; label additional failure analysis as exploratory rather than promoting every interesting subgroup into a confirmed claim.

A useful release report contains the target population, independent-unit count, trial counts, unresolved coverage, point estimate, uncertainty method, interval, decision margin, mandatory-check verdicts, and trace references. It also states what the evaluation cannot establish.

The aim is not to attach error bars to every dashboard. It is to make the strength of the release claim match the strength of the evidence. Sometimes the most informative result an agent evaluation can return is that the candidate has not earned a decision yet.