Model Swap Drill: What Breaks When You Upgrade Your Agent

A model swap is the highest-leverage change you can make to an agent system — and the most likely to silently break it. New models shift tool-call formats, change stop-condition behavior, and move costs in unpredictable directions. A model swap drill is the procedure that catches these failures before they reach production.

The takeaway in one sentence: a model swap is not an upgrade — it is a migration. Treat it like one: pin the harness, run the eval suite, diff the traces, and gate the switch on evidence.

Why model swaps break agents

The model is one component of the agent harness. But unlike tools, context, or guardrails — which you control — the model is a black box that changes its internals between versions. When you swap GPT-4 for Claude, or upgrade from one model version to the next, three things shift:

Tool-call syntax drifts. The new model may emit tool calls in a slightly different format — different key ordering, different whitespace handling, sometimes a different schema entirely if the API changed. Your tool parsers break on edge cases the old model never hit.

Stop-condition behavior changes. Models have different tendencies about when to stop: one model calls “done” after two steps; another loops three more times before converging. The agentic loop parameters that worked for the old model — max iterations, retry budgets, verifier thresholds — may be miscalibrated for the new one.

Cost and latency profiles shift. A model that generates 30% more tokens per turn is 30% more expensive per run, regardless of per-token pricing. A model that is faster per token but takes more steps to converge may cost more in total. You need the full-run economics, not the unit price.

The swap drill procedure

A swap drill is a structured test that answers one question: does the new model produce equivalent-or-better outcomes on the same tasks under the same harness? Run it before you deploy, not after.

Step 1 — pin the harness

Freeze everything except the model. Same prompts, same tools, same context strategy, same guardrails, same loop parameters. If you change two things at once, you cannot attribute a regression. This is the experimental hygiene the meta harness pillar exists to enforce.

Step 2 — run the eval suite against both models

Your eval suite is the fixed task set. Run every task against both the old model (baseline) and the new model (candidate), with identical harness configuration. Record:

• Pass/fail per task — did the agent complete the task correctly?
• Tool-call count — how many invocations per run?
• Token usage — prompt tokens, completion tokens, total cost per run.
• Steps to completion — how many loop iterations?
• Error rate — tool errors, parse errors, guardrail trips.

Step 3 — diff the traces

This is where observability pays for itself. For every task where the two models diverged — one passed, one failed, or both passed but took different paths — pull the traces and diff them:

Task: "deploy-site"
  OLD MODEL (passed, 4 steps, 2.1k tokens)
    step 1: read_file(deploy-site.sh) → 200 OK
    step 2: exec(npm run build) → success
    step 3: exec(wrangler pages deploy) → success
    step 4: verify(url) → 200 ✅

  NEW MODEL (failed, 6 steps, 3.4k tokens)
    step 1: read_file(deploy-site.sh) → 200 OK
    step 2: exec(npm run build) → success
    step 3: exec(npm run build) → skipped (duplicate) ← unnecessary retry
    step 4: exec(wrangler pages deploy --dry-run) → success ← unexpected flag
    step 5: exec(wrangler pages deploy) → success
    step 6: verify(url) → ERR: wrong URL ← context drift

The trace diff tells you exactly what changed: the new model retries unnecessarily, adds unexpected flags, and misreferences the URL. These are harness-level failures that a pass/fail number would hide.

Step 4 — check the economics

Build a simple comparison table:

Metric	Old Model	New Model	Delta
Tasks passed	47/50	44/50	-3
Avg tokens/run	2,840	3,620	+27%
Avg steps/run	4.2	5.1	+21%
Cost/run	$0.034	$0.051	+50%
Avg latency	12.4s	9.8s	-21%

In this example, the new model is faster per run but costs 50% more and fails three more tasks. That is not an upgrade — it is a regression with a speed bonus. The decision is now data-driven, not a vibe from “the new model feels smarter.”

Step 5 — gate on criteria

Define your swap criteria before you run the drill, not after. Example thresholds:

• Task pass rate: new model ≥ old model (no regressions allowed)
• Cost per run: new model ≤ old model × 1.2 (20% cost ceiling)
• Latency: new model ≤ old model × 1.5
• No new guardrail trips in the eval suite

If any threshold fails, you do not ship the swap. You either tune the harness for the new model (adjust loop parameters, update tool schemas) or you stay on the old model. The drill gives you the evidence; the criteria make the decision.

What usually breaks

Patterns that recur across model transitions:

Tool-call format changes. The most common breakage. Models change how they emit JSON arguments — quoting, key order, null handling. Fix: validate tool-call schemas on every model version, not just the one you built for.

Over-zealous or under-zealous stepping. The new model may be more thorough (more steps, more tokens, higher accuracy but higher cost) or more terse (fewer steps, misses edge cases). Fix: adjust max_iterations and retry budgets per model, not globally.

Prompt sensitivity. A prompt written for one model’s tendencies — “be concise,” “think step by step” — may land differently on a new model. Fix: run the versioned prompts through the new model and check whether the instructions still produce the intended behavior. Sometimes a prompt revision is needed; often the existing prompt is fine, but you only know if you test.

Verifier mismatch. If your verifier checks for a specific output format or keyword that the new model produces differently, the verifier fails even when the task is correct. Fix: verify that verifier assumptions hold across models.

The pre-deploy checklist

Before shipping a model swap:

• Eval suite passes at ≥ old-model rate
• Trace diff reviewed for all divergence cases
• Cost per run within threshold
• No new guardrail trips
• Tool-call schemas validated
• Observability layer logging the model version field
• Rollback path documented (switch the model config back)

Failure log

Failures from running swap drills in practice:

• First drill compared pass rates only, ignored trace diffs. Two tasks “passed” on the new model but used a completely different tool path — one that happened to work but would fail on edge cases. Trace diffs caught it on the second run.
• Trusted the model’s API changelog to list all behavioral changes. It did not mention that the new model emits tool calls with an extra whitespace field that broke our naive JSON parser. Added a schema validation step to the drill.
• Ran the drill with a 10-task subset instead of the full 50-task eval suite. The 10 tasks happened to be the ones the new model handled well. Full suite showed three regressions we would have shipped.

Takeaway

A model swap is a migration, not an upgrade. Pin the harness, run the full eval suite against both models, diff the traces, check the economics, and gate on criteria you defined before the drill. The procedure takes an afternoon. Shipping a broken model swap takes a week to detect and another week to roll back — if you even can.