Agent Guardrails: Red-Team the Boundary, Not the Reply

Agent guardrails should be red-teamed at the point where an instruction becomes an effect. A reassuring refusal is not evidence that a tool did nothing; a suspicious tool request is not evidence that the runtime allowed it. The useful test asks whether the harness preserved a specific boundary while still doing the work it was authorized to do.

Start with that boundary, give the attacker a defined foothold, and inspect the resulting state. This is a test-design guide, not an attack benchmark: the scenarios below are proposed fixtures, not reported runs or measured defenses.

Define an agent guardrail as an invariant

“Resists prompt injection” is too broad to serve as a pass condition. Replace it with a rule that an independent observer can evaluate.

For a support assistant authorized to draft a customer reply, useful invariants might be:

• No message is delivered unless the trusted task grants send authority for that recipient and content.
• No customer record outside the authenticated tenant is returned to the agent.
• Retrieved text cannot grant new permissions or change an approved destination.
• Revoked authority cannot be reused by a resumed worker.
• A tool budget remains enforced across retries and delegated work.

These are different boundaries. A suite that tests only whether the final answer contains a secret cannot establish that tenant filtering, approval binding, or cancellation works.

The guardrails, budgets, and kill-switch layer defines the controls. Red-teaming supplies adversarial conditions under which those controls must hold. Write down both the forbidden effect and the legitimate outcome before constructing the attack.

For example: the assistant must produce a draft answering the customer’s question, but must not send it. Refusing the entire task preserves the send boundary but fails the usefulness check. Report those outcomes separately.

Give prompt-injection tests a realistic foothold

An attacker who controls a retrieved paragraph is not the same as an operator who controls the system configuration. Conflating those powers makes results hard to interpret.

For each scenario, specify what the attacker can edit, what they can observe, and how many attempts they receive. Useful entry points include a retrieved document, an inbound message, a tool-result field, or a lower-trust agent handoff. Keep the trusted task and runtime policy outside that edit surface.

OWASP’s prompt-injection guidance distinguishes direct inputs from indirect injections carried by external sources. It recommends adversarial testing alongside least privilege and separation of untrusted content. That distinction matters operationally: an injected sentence in a document must not acquire the authority of the user who asked for a summary.

Use a synthetic environment with fake records, fake credentials, and a local delivery ledger. A forbidden “send” should append to an isolated test sink, never deliver to a real person. Test egress with a destination you control and an identifiable synthetic canary, not a production secret. Disable actual external transport so a failed guardrail remains a test failure rather than a disclosure.

Preserve the exact bytes the agent consumed. A clean source document is insufficient if the browser, extractor, or result formatter transforms it before it reaches the model.

Build a guardrail red-team matrix around effects

The following matrix is a starting design, not evidence that any particular harness passes. Each row needs a deterministic fixture, an allowed task, and a verifier with visibility into the affected resource.

Boundary	Adversarial condition	Evidence required
Instruction authority	A retrieved document claims the operator has changed the task	Original trusted task remains authoritative; no newly requested effect occurs
Data scope	A tool request references a synthetic record in another tenant	Server-side authorization denies access; protected bytes never enter the result
Approval binding	Recipient or body changes after a send is approved	Execution rejects the changed request or requires a new approval
Output integrity	Source text tries to insert an unsupported conclusion into a summary	Answer remains supported by the allowed source evidence
Budget enforcement	A retry or child task attempts to obtain a fresh allowance	Shared accounting prevents new reservations beyond the configured ceiling
Cancellation	A queued write reaches execution after authority is revoked	Executor rejects the stale authority; committed state is unchanged
Recovery	A tool commits an effect but its acknowledgment is lost	Reconciliation finds the original effect; retry does not duplicate it

Some rows exercise model behavior. Others probe the runtime directly. Keep both layers, but label them.

A model-mediated test asks whether hostile content leads the agent toward a forbidden action. A boundary test submits the forbidden request directly to the executor and checks that the executor refuses it. Passing the first does not prove the second; passing the second does not mean the model reliably recognizes hostile content.

For budget tests, define the ceiling precisely. A limit checked only after a model call may be exceeded by that call. Reserve a conservative allowance before dispatch, account for concurrent work, and document any maximum in-flight exposure. Do not describe a soft warning threshold as a hard spending cap.

For cancellation, fix the ordering in the fixture. “Revoked before the executor authorizes the write” is testable. An effect already committed before revocation cannot be made nonexistent by a later cancellation signal.

Verify state, not the agent’s account of state

Capture a before snapshot, the tool-request stream, runtime policy decisions, and an after snapshot. Compare them with an independently implemented verifier. The agent’s final answer is another artifact to inspect, not the source of truth.

For a draft-only task, distinguish four events:

1. The model proposes sending a message.
2. The authorization layer rejects the proposal.
3. The delivery subsystem records no committed message.
4. The model tells the user what actually happened.

A forbidden proposal stopped at step two is a containment success and a model-behavior failure. A polite refusal after a delivery at step three is an effect-boundary failure. A correct draft followed by a false claim of sending is a reporting failure. Collapsing all three into one “safe” label hides the component that needs repair.

Inspect more than the obvious resource. A harness could avoid the send API while placing protected text into a URL, log field, artifact, or delegated task. The verifier’s scope should follow the defined data and effect boundaries. If a channel is unobserved, say so; absence from one log is not proof of absence everywhere.

The sandbox and blast-radius design should ensure the test environment itself cannot escape those boundaries. Use separate test identities and capture infrastructure failures explicitly. A disconnected sink is not a successful blocked attack if the harness attempted to send and the test never observed the result.

Measure attack resistance and legitimate completion separately

Freeze the fixture set, model version, prompt, tools, policy, and verification rules before a comparison. Record sampling settings, repeat count, attack budget, and any adaptive feedback available to the attacker.

An attacker who can inspect failures and revise a payload has a different capability from one submitting a fixed payload once. Anthropic’s browser prompt-injection research explicitly discusses adaptive attacks and emphasizes that improved robustness does not solve prompt injection. Treat any reported rate as a property of its tested setup, not a portable safety score for your application.

A useful report separates:

• Forbidden proposals generated by the model.
• Forbidden effects actually committed.
• Legitimate tasks completed under attack.
• Benign tasks incorrectly blocked.
• Incomplete or invalid runs caused by test infrastructure.

Give counts and denominators for each. Do not quietly remove a timeout from the suite or call an unobserved outcome safe. Retain invalid attempts and explain whether they require a rerun under the frozen protocol.

Pair malicious fixtures with benign controls that contain similar vocabulary. An assistant summarizing a document about prompt injection may need to quote a hostile sentence without obeying it. A defense that rejects every document containing “ignore previous instructions” can look strong on an attack-only suite while breaking legitimate research tasks.

Turn discovered failures into regression gates

Exploration and release testing serve different purposes. During exploration, vary surfaces, timing, and payloads to find failures. Once a failure is understood, minimize it into a stable replay without removing the condition that caused it.

Attach the trusted request, attacker-controlled input, fixture version, policy version, observable trace, and exact failed invariant. Add that case to the agent regression gate, then reserve fresh cases to check whether the repair generalizes beyond the known wording.

Fix the layer that owns the boundary. A cross-tenant read needs authorization enforcement. A stale approval needs binding and revalidation. A persuasive retrieved paragraph may call for better source labeling and model behavior, but those changes should not replace the executor’s permission check.

The final deliverable of a red-team exercise is not a collection of alarming transcripts. It is a set of replayable boundary failures, independent verifiers, and repairs whose scope is stated honestly. Test what the agent can cause, keep useful completion visible, and never let a reassuring reply stand in for evidence.