Guardrails, Budgets, and Kill Switches: Designing Agents for Failure

Your agent will fail. Not might — will. The question is whether it fails downward (stops, reports, waits for a human) or fails sideways (burns tokens in a loop, emails the wrong person, deletes the wrong table). Guardrails are the difference. Everything in this post assumes you’ve already built the loop and the tool surface; this is the layer that makes both survivable.

Three limits, one idea

A guardrail is a limit the harness enforces that the model cannot argue its way past. There are exactly three kinds worth engineering, and they compose:

Budgets: the runtime wins, not the prompt

The most common guardrail mistake is writing it into the prompt. “Do not use more than 10 tool calls” is a suggestion. The model will honor it right up until it doesn’t — usually under load, on a hard task, exactly when you need it most.

Enforce budgets in the harness:

• Token budget. Count input tokens per turn. Past the cap, force compaction or terminate.
• Time budget. A wall-clock deadline checked every loop iteration, not an estimate.
• Money budget. If tools cost money (APIs, compute, sandboxes), meter it. Kill at the ceiling.
• Iteration budget. Max N loop turns, full stop. If the task genuinely needs more, that’s a design signal, not an argument for a higher cap.

The interesting design question is what happens at the boundary. A good harness doesn’t just die — it checkpoints state, writes a failure record, and exits in a resumable way. That recovery contract depends on idempotency: a rerun must recognize work already completed rather than repeat its external effects.

Permission scoping: minimize blast radius

Think about each tool as an attack surface, because it is one — the model isn’t adversarial, but it is indifferent: it will happily call the delete endpoint if that’s the shortest path it sees to the goal.

Practical tiers:

1. Read-only tools — no approval needed. The worst outcome is a leaked read, and you control the source list.
2. Reversible writes — allowed with logging. Creating a draft, staging a file, queueing a message.
3. Irreversible or external effects — sending, publishing, deleting, paying. Route through a human, or gate behind an allowlist so tight it hurts. A useful escalation design defines the trigger, the evidence packet, and the safe state while approval is pending.

The discipline that makes this work is refusing the temptation to grant “just this once” exceptions in config. If the agent needs write access to ship, build a narrow write tool that can only do the shipping thing. Don’t hand it a shell and a prayer — tool design is guardrail design wearing a different hat.

Kill switches: stopping is a feature

A kill switch is the operator’s override: stop the agent now, preserve state, tell me what it was doing. It has three requirements people routinely miss:

• Reachable from outside the agent. If the only way to stop the agent is a tool the agent itself controls, you don’t have a kill switch — it does.
• Fast. Budgets check per-iteration; the kill switch should interrupt mid-tool-call if possible. Waiting for the current “just one more search” turn to finish is how a 30-second incident becomes an hour.
• Auditable. Killing the agent must leave a trace: what step it died on, what side effects already landed. Without that, recovery is archaeology.

"Most agent failures aren't model failures. They're loop failures — and loop failures are guardrail failures you didn't build."

Circuit breakers the model can’t see past

Behavioral tripwires catch what budgets can’t — the run that’s cheap but wrong. The three that earn their keep:

• Repetition detection. Same tool, same args, N times in a row → trip. This is the single highest-value breaker in practice; loops are the most common agent failure mode.
• Error-rate threshold. K consecutive tool errors → stop and escalate instead of letting the model “try a different approach” for the fifteenth time.
• Output validation. Schema-check every tool output. A malformed payload that passes silently poisons context and every downstream decision — fail the run, don’t hope.

Each breaker should write a structured incident record: trigger, step index, recent tool calls. You’ll use those records to tune thresholds, and they double as training data for your evals. Those incident records are also the raw material for your observability layer — the harness events that tell you which guardrails are firing and why.

What this doesn’t solve

Guardrails make failure cheap; they don’t make the agent correct. An agent inside perfect guardrails can still confidently produce the wrong answer on budget and on time — that’s a verifier problem, and it’s the next experiment in this series. Guardrails bound the downside. Verifiers raise the mean. You need both, and confusing them is why so many “safe” agents are safely useless.

Start small: an iteration cap, a repetition breaker, and read-only defaults will eliminate the majority of catastrophic agent failures this week. Add the rest when the incident records tell you to.

Takeaway

Build the guardrails before you need them, enforce them in the runtime instead of the prompt, and make stopping cheap enough that you never hesitate to do it. The harness is where reliability is won — the guardrail layer is where catastrophe is prevented.