Agent Concurrency Control: Stop Stale Workers

Agent concurrency control is not a promise that only one process is running. A timed-out worker can wake up after its replacement starts. A delayed tool request can arrive after ownership changes. Two independent tasks can target the same document without sharing a task identifier.

The takeaway: enforce authority and freshness at the boundary that commits the change, not only when a worker starts. Use revision checks for stale inputs, leases for recoverable ownership, and fencing for stale owners. These mechanisms solve different problems.

This is a design guide. The timelines and acceptance cases below are illustrative, not observations from a completed experiment.

Agent concurrency starts with the protected resource

Imagine an agent that publishes a documentation update. Worker A reads the current article, prepares a replacement, and pauses. Worker B takes over and publishes a newer candidate. A resumes and uploads its earlier draft. Both workers may have passed a build. Only one candidate should become the current publication.

Before choosing a lock library, identify the invariant: a stale owner must not replace a publication accepted under newer authority. Then identify the actual resource: the tenant’s article or deployment target, not merely A’s queue message.

A task-scoped lock cannot prevent a separate task from changing that same article. Conversely, one global lock can unnecessarily serialize independent tenants and unrelated documents. Define ownership and revision scope around the resource whose state could conflict, and make that scope explicit in the write contract.

The production agent architecture guide separates queue delivery, durable state, executors, and verification. Concurrency crosses those boundaries: the scheduler can assign work correctly while an unguarded storage adapter still accepts an obsolete write.

Distinguish duplicate preparation from conflicting commitment. It may be acceptable for two workers to search documentation or build isolated candidates. It is different for both to change a shared publication pointer. Put the strongest coordination at that commitment boundary; do not hold a database transaction open while a model spends minutes drafting.

Revision checks reject stale agent decisions

A revision check answers: does the state still match the version this proposed change depends on?

Have the worker retain the authoritative input revision with its candidate. At commitment, compare that expected revision with the current one and apply the update atomically only if they match. A read followed by an unconditional write is not a revision check, even if the read happened immediately before the write.

For an illustrative resource at revision 12, two candidates may both expect revision 12. Once one commits and advances the revision, the other must fail its comparison. It should reload current evidence and decide whether to recompute, merge under explicit rules, or stop. Blindly replacing its expected revision with the latest number defeats the protection: its reasoning still used old input.

Use a revision that cannot silently repeat during the relevant lifecycle. A value changing from A to B and back to A can fool a value-only comparison when the intervening history matters. Likewise, deleting and recreating a resource should not make an old candidate appear current. Include resource incarnation or another durable version identity where necessary.

If a decision depends on several records, comparing only the output document’s revision may be insufficient. A policy or approval could change without touching that document. Validate the relevant dependencies under a transaction or an explicitly versioned commitment contract.

The etcd API guarantees describe atomic, strictly serializable KV operations and increasing modification revisions. They also distinguish those guarantees from watch behavior and potentially stale serializable reads. That distinction matters: a watch notification can trigger work, but it is not a substitute for a transactional condition at commitment. Whatever store you choose, verify its actual transaction and consistency contract.

Worker leases allow recovery, not remote termination

A lease answers: which attempt may currently renew its claim, and when may another attempt try to take over?

Acquisition, renewal, and replacement need authoritative conditional transitions. A renewal should identify the owner and generation it intends to extend. Otherwise, an old worker could accidentally renew or overwrite its replacement’s claim. A release must likewise be conditional; an unconditional delete can remove someone else’s ownership record.

Choose time and expiry semantics deliberately. Let the coordination service enforce its lease contract rather than trusting each worker’s wall clock as independent proof of authority. A local timeout should cause the worker to stop issuing new work, but local cooperation is not the final correctness boundary.

Expiry makes replacement possible. It does not kill a process, retract a packet, or cancel an external request. A runtime pause, partition, or delayed request can leave two workers believing they are entitled to act. Increasing the timeout may reduce how often this happens; it does not prove it cannot happen.

Lease duration is therefore an operational choice involving recovery delay, renewal traffic, and expected pauses. Correctness must survive an unfortunate timing choice. If the ownership service is unavailable, a correctness-critical path should not silently substitute an unverified cached claim for the required check.

Fencing tokens make stale ownership enforceable

Fencing gives the protected resource a way to reject requests from older ownership generations.

An ownership authority issues a monotonically increasing token for each new acquisition within the protected scope. The worker carries that token to the write boundary. The receiver durably tracks the newest accepted generation and checks the token atomically with the mutation. A smaller generation cannot overwrite work already accepted under a larger one.

Martin Kleppmann’s distributed-locking analysis explains this through a paused lease holder whose delayed write arrives after a newer holder’s write. The storage service must participate. Adding a token to a trace, prompt, or HTTP header that the receiver ignores provides no fencing.

There is an important limit: a highest-seen-token check rejects the old holder only after the receiver has learned the newer generation. Suppose generation 42 is granted, but the receiver still knows only 41. A delayed request carrying 41 may arrive first. If the contract requires rejection immediately after reassignment, establish the new fence at the resource as part of the takeover protocol before treating takeover as complete, or validate current authority atomically at the commitment boundary.

Do not describe these two contracts as equivalent. Rejecting writes older than the newest accepted fence protects ordering; rejecting every write after lease expiry or reassignment requires additional coordination with current authority.

The generation must remain ordered across restarts and resource recreation. A random worker UUID identifies an attempt but cannot tell the receiver which attempt is newer. Timestamps inherit clock assumptions. Use an authority with a documented durable ordering guarantee, and define how its tokens map to the resource being protected.

Fencing also does not order every operation within one generation. If a worker can issue concurrent writes under the same token, use resource revisions or an operation sequence where ordering matters.

Mechanism	Question it answers	What it does not establish
Revision comparison	Is this candidate based on the current protected state?	This worker still owns execution
Lease	When can another attempt claim recoverable work?	The previous process has stopped
Fencing token	Is this request older than the enforced ownership generation?	The request is not a duplicate within that generation
Idempotency key	Is this another delivery of the same logical effect?	A different effect is authorized or based on fresh input

External agent tools need a real dispatch boundary

Many external APIs do not understand your fencing token. A tool gateway can mediate access, but its own check must not leave the same race in a different process.

Consider a gateway that reads owner 42, pauses, then sends a request after ownership changes to 43. Checking ownership before dispatch was useful, but not atomic with dispatch. Do not claim it prevents all effects from an old generation.

One practical contract is a durable dispatch commitment. In a transaction, validate current task status, owner, required revisions, and effect identity; record a dispatch intent with an exact payload. A dispatcher then owns delivery of that committed intent. Replacement workers reconcile it instead of creating a competing effect. Protect the dispatcher and its shared ledger against duplicate or competing delivery too; moving the boundary does not remove the problem.

Under this contract, cancellation or reassignment blocks new commitments. An already committed intent may still execute later. If the product promises stronger cancellation, the target must support it, or the architecture needs a stronger serialized effect path and a defined way to resolve in-flight work. Do not offer an impossible guarantee through a status flag.

Retries of one committed intent should retain its logical effect identity across worker generations. Generating a fresh idempotency key on takeover can turn a recovery attempt into a second publication or notification. The idempotency guide for agent actions covers stable intent keys and reconciliation. Fencing and deduplication complement each other; neither replaces the other.

Preserve unknown outcomes after timeouts. A missing reply is not permission to assume the old effect failed, and a newer ownership token does not undo an action the provider already accepted.

Test competing workers at controlled boundaries

Test the invariant by controlling the interleaving, not by hoping two fast threads collide. Pause a worker after reading state, after checking authority, or after sending an effect. Resume it only after the competing transition has committed. Use a controllable clock or lease fixture where possible; wall-clock sleeps make races harder to reproduce.

The following are proposed acceptance cases, not passing test results:

Controlled interleaving	Required observation
Two candidates read the same revision; one commits first	The second comparison fails without changing current state
A pauses; B takes over and establishes the newer resource fence; A resumes	A’s older token cannot mutate the protected resource
A delayed old token arrives before the receiver learns the new generation	Behavior matches the documented takeover contract, including any fence-establishment step
A stale worker renews or releases after replacement	The replacement’s ownership record remains intact
Two distinct tasks target the same article	Resource-scoped revision or ownership checks prevent a lost update
The provider accepts an effect but the reply is lost	Recovery resolves the original intent rather than silently creating a second one
Cancellation races with dispatch commitment	The recorded commitment order determines whether delivery remains allowed
A resource is deleted and recreated while an old candidate waits	Old-incarnation evidence cannot authorize the new resource’s mutation

Inspect authoritative records and the effect target. A worker logging “stale write rejected” is insufficient if the document changed anyway. Retain task and resource identities, input revisions, generations, effect keys, commitment references, and rejection reasons so the failure is reconstructable without copying secrets into logs.

Add a progress check as well: once the current owner is healthy and its dependencies are current, it should eventually be able to commit. A design that rejects every write is safe only in the narrowest sense; it does not complete the user’s task.

Start with one protected effect. Specify its revision contract, takeover semantics, and dispatch commitment. Then pause the old worker at the worst possible point and prove that the resource still obeys those rules. Concurrency control belongs where the change becomes real.