Agent Recovery Checkpoints: Resume Without Repeating

Agent recovery checkpoints should preserve enough evidence to choose the next safe action, not merely enough conversation to continue generating text. Resume from durable task state, reconcile uncertain external effects, and revalidate authority before dispatching new work. A fluent summary of the previous run cannot establish whether its last tool call actually committed.

This is a design guide, not a benchmark or an incident report. The records and failure cases below are illustrative. The goal is a restart contract that survives an interrupted worker without silently repeating its effects or forgetting its constraints.

What an agent checkpoint must preserve

A conversation transcript records what participants said. A recovery checkpoint records what the system can establish and what work remains permissible. Those records overlap, but they serve different purposes.

Suppose an agent prepares a support ticket, sends the create request, and loses its connection. Its last assistant message might say “creating the ticket.” That text does not distinguish a request that never left the machine from a ticket created successfully without a returned receipt.

A useful checkpoint separates several kinds of state:

State	What recovery needs
Task intent	Stable task identity, requested outcome, scope, and acceptance criteria
Execution position	Next eligible step, dependencies, and the revision of the workflow
Prepared effects	Stable operation identities, immutable payload references, and payload digests
Observed outcomes	Pending, unknown, applied, rejected, or partially applied effects, with evidence
Accepted work	Verified artifacts, their versions, and the checks that accepted them
Authority and limits	Approval references, cancellation status, deadline, and remaining budgets
Recovery ownership	State revision and the current worker’s right to advance it

Treat this as an application contract, not a universal framework schema. A read-only research workflow needs less effect bookkeeping than a workflow that sends messages or changes production configuration.

LangGraph’s persistence documentation makes a useful distinction between thread-scoped graph checkpoints and cross-thread stores. It also notes that in-memory checkpointers lose their contents on process restart. A configured checkpointer is not automatically durable storage; choose and operate the backing store for the failure domain you need to survive.

Place durable checkpoints around effect boundaries

Saving every few messages is a scheduling policy, not an effect-safety policy. The critical boundary is where computation becomes an external change.

Before dispatch, persist the intended operation and its exact payload. Do not let a replacement worker regenerate either from a summary. If preparing the payload involved nondeterministic model output, preserve the accepted artifact rather than assuming another generation will reproduce it.

Use an explicit dispatch protocol. A worker first records prepared intent. Immediately before attempting the external call, it durably advances the operation into a state that means dispatch may have occurred. Only then does it call the tool. On restart, that state is unresolved until the owner of the effect supplies evidence.

This ordering deliberately admits a conservative case: the worker may crash after recording possible dispatch but before sending anything. Recovery still has to reconcile it. The opposite ordering risks an external write with no durable record that it was attempted.

After receiving a result, persist the receipt and outcome evidence. After verification, persist the accepted result and advance the workflow. Where those records share a transactional database, commit related state transitions together. Where they do not, keep the intermediate states recoverable.

There is still a gap between a remote service committing a write and the harness saving its receipt. A local checkpoint cannot close that gap by itself. You need the receiver’s deduplication contract, an authoritative operation lookup, or a controlled unresolved state. This is why idempotency for agent reruns belongs beside checkpointing, not underneath a vague “retry on failure” setting.

A recovery record should route the next action

The following JSON is an illustrative application record, not a framework configuration or a complete storage schema. Its referenced records would hold the full payload, authority, limits, and evidence:

{
  "schema_version": 1,
  "task_id": "case-8472-follow-up",
  "workflow_version": "support-follow-up-v3",
  "state_revision": 12,
  "phase": "reconcile_effect",
  "operation_id": "ticket:create:case-8472",
  "payload_ref": "artifacts/case-8472/ticket-v1.json",
  "effect_status": "unknown",
  "receipt_ref": null,
  "authorization_ref": "approvals/case-8472/revision-4",
  "limits_ref": "limits/case-8472/current",
  "evidence_refs": ["events/case-8472/dispatch-1"]
}

The important field is not the prose description of what happened. It is the combination of a stable operation, retained payload, explicit uncertainty, and a recovery phase that forbids blind redispatch.

References must resolve to retained, access-controlled records. A payload digest can detect substitution; it cannot recover a deleted payload. A checkpoint pointing to an expired temporary file is an incomplete recovery package.

Keep credentials out of the record. Persist scoped identity and approval references, then obtain currently authorized access through the normal credential path. Treat checkpoint content and tool evidence as data: neither can issue itself new permissions.

Resume interrupted agents with a decision procedure

A restarted worker should not begin by asking a model what it remembers. It should load the durable state and establish which transition is allowed.

First, validate the checkpoint schema, workflow compatibility, and required artifacts. An unreadable checkpoint is a block to diagnose, not permission to restart the business operation from scratch. Preserve the original record if a migration is necessary.

Next, acquire recovery ownership through a conditional state transition. Check cancellation, the task deadline, remaining budgets, and current authority. A process restart must not reset a spent retry allowance or turn a cancelled task back into an active one.

Then classify the pending work:

Durable evidence	Allowed recovery direction
Pure computation incomplete	Recompute from retained inputs, subject to budgets
Prepared effect never eligible for dispatch	Revalidate inputs and authority before dispatch
Dispatch possible; terminal outcome absent	Reconcile against the effect owner
Effect applied; acceptance not established	Run the verifier, without automatically repeating the write
Accepted result and valid evidence retained	Continue from that result without regenerating it
Partial batch outcome	Recover each unresolved operation independently
Missing evidence or incompatible state	Preserve the block and escalate with the recovery record

The “never eligible” row is justified only if the dispatch protocol enforces that property. Absence of a log line is not equivalent evidence.

Our agent tool call error guide explains the classification behind this procedure: request validity, execution outcome, and verified completion are separate facts. A verifier timeout should not become another create request.

Reconciliation has a retention window

Recovery can happen much later than a normal retry. That changes the assumptions available to the harness.

Stripe’s idempotent-request documentation provides a concrete example: it documents parameter matching, saved results, and the possibility of pruning keys after they are at least 24 hours old. Reusing a key after its original record is pruned generates a new request. These are Stripe’s documented semantics, not a guarantee every tool shares.

An agent paused for days therefore cannot infer safety merely because it still has the old key. Its checkpoint retention, maximum interruption duration, and the receiver’s deduplication retention must fit together. If the guarantee may have expired, find the original operation through authoritative evidence or stop for reconciliation; do not mint a fresh key to get past uncertainty.

A search index returning no result may lag behind the actual write. A local “failed” flag may describe transport failure rather than the external operation. Recovery needs evidence from the system responsible for the effect, with the consistency limitations made explicit.

Recovery ownership must survive overlapping workers

A replacement worker can start while the old worker is merely slow. Saving a checkpoint does not prevent both from continuing.

Use conditional updates against the expected state revision so stale workers cannot overwrite newer progress. For external effects, enforce ownership at the executor or receiving boundary too. A database lease that expires locally does not make a paused worker forget its credentials or cancel a request already in flight.

Where supported, use a monotonically increasing fencing token that the effect boundary checks, or route writes through an executor that enforces the current owner. For a service without fencing support, combine enforced dispatch ownership with receiver-side idempotency and explicit reconciliation of in-flight requests. Do not describe a lease alone as exactly-once execution.

Recheck human approval against the actual payload and target state. If either changed, the old approval may no longer apply. Recovery preserves authorized intent; it does not extend authority indefinitely.

Test crash recovery at the dangerous boundaries

A graceful shutdown test is useful but insufficient. The following are proposed failure-injection cases, not reported experiment results:

• Stop after prepared intent is persisted but before dispatch becomes eligible; verify that the original payload and operation identity survive.
• Stop after possible dispatch is recorded but before the call; verify that recovery resolves uncertainty without inventing success.
• Commit the external effect and drop its response; verify that resumption does not produce a duplicate effect.
• Stop after receipt persistence but before verification; verify that the checker runs without repeating the write.
• Resume two workers from the same revision; verify that stale progress and unauthorized dispatch are rejected.
• Resume after cancellation, approval expiry, or deduplication expiry; verify that recovery respects the changed boundary.
• Remove a referenced artifact or change the workflow version; verify that incompatibility produces a visible block.

Assert against external state and durable records, not only the final assistant response. Count duplicate effects, lost accepted work, unresolved operations, and exhausted budgets alongside eventual completion. Convert important failures into behavior-verified agent regression gates so recovery remains part of the release contract.

Takeaway

A good checkpoint answers three questions: what is durably known, what remains uncertain, and what is still authorized. Preserve intent before effects, keep uncertain dispatches explicit, and accept outcomes only with evidence. The purpose of resumption is not to continue the conversation. It is to continue the work safely.