The Verifier Spectrum: From Schema to Ground Truth

Every agent step produces output, and someone has to check it. That someone is a verifier — code that runs after the model answers and decides whether the answer is good enough. But not all verifiers are equal. A JSON schema check costs milliseconds and catches nothing about whether the content is correct. A human review catches everything but defeats the purpose of automation. Between those poles lies the entire practice of agent verification, and the skill is knowing which level to apply where.

The six levels

Verifiers exist on a spectrum from cheap-and-fuzzy to expensive-and-exact. Each level catches more failure modes than the one below it, at higher cost.

Level 0 — Schema validation

Did the model produce well-formed output? Is it valid JSON? Do the required fields exist? Are the types right?

def verify_schema(output: str) -> bool:
    try:
        data = json.loads(output)
        return all(k in data for k in REQUIRED_FIELDS)
    except json.JSONDecodeError:
        return False

This costs nothing and catches the most common LLM failure mode: the model didn’t follow the output format. It catches nothing about content quality. A response with the right structure and the wrong answer passes. Schema validation is table stakes — every step should have it — but it is never sufficient on its own.

Level 1 — Constraint checks

Beyond structure, does the output satisfy the rules of the task? Word count within bounds. URLs resolve. Code compiles. Dates are in the future. Enumerated values come from the allowed set.

def verify_constraints(data: dict) -> list[str]:
    violations = []
    if not (50 <= len(data["summary"]) <= 200):
        violations.append("summary length out of range")
    if data["action"] not in ALLOWED_ACTIONS:
        violations.append(f"action '{data['action']}' not permitted")
    return violations

Constraint checks are still cheap and fully deterministic. They catch a lot: models routinely produce summaries that are too long, actions that aren’t in the playbook, dates in the past. But they cannot catch semantic errors — a summary that is the right length but says the wrong thing sails through.

Level 2 — Self-consistency

Ask the model again. Does a second pass produce the same answer? Run the agentic loop twice with identical inputs and compare the outputs. If they diverge, something is unstable.

Self-consistency is the first level that touches meaning rather than structure. It catches hallucination patterns where the model confidently invents different facts on different runs. The cost is a second inference call per step, which doubles your latency and token budget for verified steps. Use it when the task is high-stakes and the answer space is small enough that “same answer twice” is a meaningful signal.

The failure mode: a model can be consistently wrong. If the prompt biases the model toward answer X, both passes produce X. Self-consistency catches instability, not systematic error.

Level 3 — Tool-based verification

Run the output through an external tool that does not depend on the model. If the agent wrote code, compile it. If the agent wrote a test suite, run it. If the agent claimed a URL exists, fetch it.

def verify_with_tools(data: dict) -> bool:
    if data["action"] == "deploy":
        result = subprocess.run(
            ["npm", "run", "build", "--", "--dry-run"],
            cwd=PROJECT_DIR,
            capture_output=True,
            timeout=30,
        )
        return result.returncode == 0
    return True

This is where verification starts catching semantic errors. A build that compiles has correct syntax and at least plausibly correct logic. A test suite that passes means the code does what the tests claim. Tool-based verifiers are domain-specific — there is no general-purpose “correctness checker” — but for the domains they cover, they are the most cost-effective verifier in the stack.

The guardrail layer in your harness should wire these in as stop conditions: if the post-deploy verification fails, the loop retries or escalates; if it passes, the task is done.

Level 4 — Ground truth comparison

When you have a known-good answer, compare the model’s output directly. This is what evals do: each eval case has an expected output, and the verifier scores the model’s output against it.

Ground truth is the gold standard. It catches exactly the errors you have seen before, which makes it the strongest automatic verifier and also the narrowest: it cannot catch errors you haven’t seen. Ground truth comparison is only available when you have labels, which makes it expensive to set up and limited in coverage.

The practical pattern: accumulate failure traces from observability and convert them into ground-truth eval cases. Each production failure becomes a permanent regression test. Your ground-truth verifier grows over time, but it starts narrow.

Level 5 — Human review

A person reads the output and decides. The strongest verifier. The slowest, the most expensive, and the least scalable. Human review is correct for high-stakes, low-frequency decisions: should we ship this? Should we talk to the customer? Is this response safe to send?

Anything you can identify as “I need a human to look at this” is a candidate for escalation design. Over time, the goal is to convert recurring human reviews into automated verifiers at level 3 or 4. If the human always checks the same thing, that check can be encoded.

Matching verifier to risk

Not every step needs level 4 verification. The principle is simple: match the verifier level to the blast radius of a wrong answer.

Step type	Blast radius	Recommended level
Formatting output	Low — cosmetic	0 (schema)
Internal routing decision	Low — reversible	1 (constraints)
Generating content for user	Medium — visible	2-3 (consistency + tool)
Code generation	High — runs in prod	3 (compile + test)
Deploy / publish / send	High — hard to undo	4-5 (ground truth or human)

The mistake most teams make is applying the same verifier everywhere: schema checks on everything (too weak for high-stakes steps) or human review on everything (too slow to scale). The harness should configure per-step verifiers the same way it configures per-step tool permissions — based on what that step does.

Context engineering helps here: if the model has the right context, lower-level verifiers suffice because the model is less likely to be wrong in the first place. Verification is a safety net, not a substitute for good context.

Composition, not substitution

Real systems stack verifiers. A code-generation step runs schema validation (level 0), constraint checks (level 1: does the file path exist? is the function name valid?), compilation (level 3), and a ground-truth regression suite (level 4). If all pass, the answer is strong. If compilation fails, the loop retries. If the regression suite fails, the loop escalates.

The stacking is additive: each level catches what the one below missed. Schema validation catches format errors. Constraints catch rule violations. Compilation catches syntax and type errors. Tests catch logic errors. Ground truth catches regressions. Human review catches everything else.

The verifier you omit is the failure mode you accept. You will not catch every error — the goal is to catch the ones that matter, with the budget you have, at each step.

The verifier-coverage gap

One metric worth tracking is verifier coverage: the percentage of agent steps that have a verifier at or above the level their blast radius requires. If your deploy step has only schema validation, that is a coverage gap. If your formatting step has ground-truth comparison, that is over-verification — wasted budget.

Measure coverage the same way you measure test coverage: mapping requirements to actuals and flagging gaps. A step with no verifier is not “unverified” — it is “verified by the model’s confidence,” which is the weakest verifier of all.

Common failure modes

Verifier too weak. The classic. A prompt versioning step that only checks JSON structure passes an output with the wrong version number. Fix: add constraint checks for the specific fields that matter.

Verifier too strong for the step. Running a full regression suite on a step that reformats a date. The suite takes minutes; the step takes seconds. Fix: drop to schema or constraint verification.

Verifier and model share blind spots. A ground-truth eval built from past failures misses new failure modes. Self-consistency misses systematic bias. Fix: layer verifiers with different blind spots — tool-based checks alongside self-consistency, not instead of it.

Verifier that never fails. If your verifier has a 0% failure rate over hundreds of runs, it is either perfect (unlikely) or not testing anything meaningful. Check whether it would catch a deliberately wrong output. If it would not, it is decorative.

Takeaway

Verifiers run on a spectrum from schema checks to ground truth. Each level catches what the one below misses, at higher cost. Stack them per step based on blast radius. Track verifier coverage as a metric. The verifier you omit is the failure you accept — and the one you will debug at 2 a.m.