Agent Retry Policies: Diversify, Don't Repeat

An agent fails. It retries. Same prompt, same context, same tools, same model. It fails again — the same way. This is the most common retry bug in agentic systems, and it’s not a model problem. It’s a harness problem.

The agentic loop includes retry as a basic primitive: if the step fails, try again. But “try again” is underspecified. Retry what? With what changed? A retry that changes nothing is a loop in the strictest sense — it will produce the same output for the same input every time. The model is not a random oracle. Given identical context, it converges on the same path.

The fix: diversified retry. Each retry should change something — the prompt, the context, the tools, the decomposition, or the model — so that the next attempt is not the same attempt.

Why identical retries fail

LLM outputs are more deterministic than people think. At temperature 0, the same input produces the same output every time. At temperature 0.7, the output varies — but the failure mode often doesn’t. If the model misunderstands the task, sampling variation won’t fix the misunderstanding. If the context is missing a critical detail, stochastic re-sampling won’t conjure it.

This means a naive retry loop — “try up to 3 times” — is often worse than no retry at all. It burns tokens, adds latency, and produces the same wrong answer. The guardrail budget depletes with each attempt, and the verifier rejects each one, until the loop gives up having learned nothing.

The diversification axis

A retry policy should specify what changes at each attempt. Here are the axes you can diversify along, from cheapest to most expensive.

Add error context

Tell the model what went wrong. “Your previous output failed schema validation: the customer_id field was a string, but it must be an integer. Try again.”

This is the single most effective retry strategy for tool-use and structured-output failures. The error message from the verifier becomes input for the next attempt. The model now knows what to fix. Without this feedback, it is guessing blind.

Rephrase the prompt

Rewrite the instruction in different words. If the first attempt said “Generate a SQL query to find the top 10 customers by revenue,” the retry says “Write SQL that ranks customers by total revenue and returns the top 10.”

This works when the failure is a comprehension issue — the model parsed the instruction differently than intended. Rephrasing shifts the semantic framing and can unlock a correct interpretation. It costs nothing but a prompt template.

Simplify the task

If the model can’t solve the task in one step, decompose. A retry that says “First, identify the table that contains revenue data. Then, write a query that selects and ranks from that table” turns one failed attempt into two lower-risk attempts.

This works when the failure is a complexity problem — the task has too many steps for the model to hold in one inference call. Context engineering tells us that the bottleneck is often working memory, not model capability. Decomposition reduces the working-memory load per step.

Change the context

Add or remove context. If the failure was a hallucination about a missing API, add the API docs to the context window. If the failure was confusion from too many irrelevant context fragments, remove them. The retry policy should specify: on failure, expand context with the error explanation, or contract context to the minimal relevant subset.

Change the tool surface

If the model called the wrong tool or used the wrong parameters, restrict the tool set on retry. A retry that says “You have one tool available: execute_sql. Write the query as a string and call it” removes the ambient confusion of a broader tool set.

Conversely, if the failure was because the model lacked a tool it needed, add one. The retry policy can escalate tool access — grant the agent a capability it didn’t have on the first attempt. This should be explicit and budgeted, not an open-ended expansion.

Change the model

The nuclear option. If the task is hitting the ceiling of the current model’s capability, retrying with a stronger model is a legitimate strategy. This is expensive — both in cost and latency — and should be gated behind a budget. But a model that lacks the reasoning depth for a task will not gain it through rephrasing.

This is the retry equivalent of the model swap drill: you are making a migration decision under time pressure. Keep it in the policy so the harness can invoke it automatically when cheaper retries are exhausted.

Building a retry policy

A retry policy is not “try 3 times.” It is a sequence of escalating interventions. Here is a practical template:

RETRY_POLICY = [
    {"attempt": 1, "action": "add_error_context"},
    {"attempt": 2, "action": "rephrase_prompt"},
    {"attempt": 3, "action": "add_error_context + simplify_decomposition"},
    {"attempt": 4, "action": "restrict_tools + add_error_context"},
    {"attempt": 5, "action": "escalate_model"},
]
MAX_RETRIES = 5

Each attempt changes something. The policy escalates: cheap interventions first, expensive ones last. After MAX_RETRIES, the harness escalates to a human — the retry budget is exhausted, and further attempts are budget burning.

Retry vs. backoff

There are two reasons to retry, and the strategies are different.

Transient failure — rate limit, network timeout, service unavailable. Apply exponential backoff. Wait and try the same thing, because the failure was not about the input. Backoff gives the downstream system time to recover.

Correctness failure — wrong answer, malformed output, tool called with bad parameters. Diversify. Do not wait. Change the input, because the failure is about the input. Retrying the same prompt after a 2-second delay will produce the same wrong answer, just later.

Mixing these up is a common bug: applying backoff to correctness failures (giving the model a “rest” before asking the same question again) or diversifying on transient failures (rephrasing a query that only failed because the API was rate-limiting). Your retry policy should check the failure type first and route accordingly.

def retry_step(failure, attempt):
    if failure.type == "transient":
        return backoff_and_retry(attempt)
    elif failure.type == "correctness":
        return diversify_and_retry(attempt)
    else:
        escalate(failure)

What to log on each retry

Each retry attempt is data. Your observability layer should record:

• What the failure was — the full verifier output or error message
• What the retry changed — prompt diff, context diff, tool set diff
• Whether the retry succeeded

Over time, this data tells you which diversification strategies work for which failure types. “Rephrase” might win on comprehension failures. “Add error context” might win on schema violations. “Simplify” might win on complexity failures. The policy improves with evals built from these traces — each production retry becomes a test case for the next policy revision.

Common failure modes

Retry without feedback. The model fails, and the retry prompt is identical to the original. No error context, no rephrasing. This is the default behavior in most agent frameworks, and it is a token sink. Fix: wire the verifier output into the retry context.

Infinite escalation. Each retry adds more context, more tools, more instructions. By attempt 5, the context window is bloated and the model is overwhelmed. Fix: budget the total context growth per retry chain. If you add context on one retry, remove something else on the next.

Retry as a substitute for fixing the prompt. If the same failure happens on every run, the prompt is wrong — not the retry policy. A retry policy patches transient failures; it cannot fix a systematically broken prompt. If your retry rate for a given step is above 30%, the prompt or task decomposition needs rework, not more retries.

No retry budget. The agent retries forever, quietly burning tokens until a kill switch trips. Fix: hard cap on retries per step, with escalation to a human or a fallback path.

Takeaway

A retry that changes nothing will produce nothing new. Diversified retry policies — rephrase, add error context, decompose, adjust context, restrict tools, escalate model — turn each failed attempt into information that improves the next one. The retry that matters is not the one that tries harder. It’s the one that tries differently.