Production Agent Architecture: Four Boundaries That Matter

Production agent architecture becomes clearer when you stop treating the model loop as the whole system. A loop decides what to try. The surrounding system decides whether work exists, who owns it, what effects are allowed, and what evidence makes it complete.

The takeaway: separate work delivery, durable task state, execution authority, and outcome verification. These are responsibility boundaries, not a requirement to deploy four services. One process can implement them; four services can still blur them.

This is a design guide, not a benchmark or a report of production incidents. The examples and failure checks below are proposed cases for evaluating an implementation.

Agent work queues deliver opportunities, not truth

Imagine an agent that prepares and publishes a documentation update. The request enters a queue, a worker edits files, a build passes, and a deployment starts. If the worker disappears after the deployment service accepts the request, a second delivery must not blindly repeat the publication.

The queue knows that a message needs handling. It does not necessarily know whether an external action finished.

Amazon SQS documents at-least-once delivery: a message copy can be received again, and applications should be idempotent. That is one concrete service contract, not a claim that every broker behaves identically. Read the delivery, visibility, ordering, and retention guarantees of your actual queue.

Design the message around a stable task identity and enough routing information to locate authoritative state. Avoid treating a large serialized conversation as the latest task record. A delayed message can carry an old policy, an expired approval, or instructions for work that has already been cancelled.

On delivery, the worker should load current state and claim an eligible attempt through an atomic transition. A duplicate delivery for completed work becomes a harmless lookup. A delivery for work already owned does not automatically authorize another executor. A queued message that arrives after cancellation cannot resurrect the task.

Acknowledgement means responsibility has moved to a durable place. If the worker acknowledges before finishing, some durable scheduler must still own recovery of unfinished work. If it acknowledges afterward, redelivery must remain safe. Choose the policy explicitly; do not let a client library default define your recovery model.

Durable agent state connects admission to recovery

Persist the accepted task contract before dispatching work. The record should identify the requested outcome, caller and tenant, permitted effects, deadline, budget, input revision, and acceptance criteria. Admission also needs deduplication: if a client retries a timed-out submission, it should be able to recover the original task instead of accidentally creating a second one.

Then address the gap between recording the task and notifying a worker. A database commit can succeed while a queue publish fails. Publishing first reverses the problem: a worker may receive a task that was never committed.

The transactional outbox pattern records the state change and an event in the same database transaction. A separate relay publishes committed events. The relay can still publish a duplicate, so consumers still need deduplication. The outbox connects database commitment to recoverable dispatch; it does not make an unrelated external API call atomic with the database.

The following responsibility map is useful even inside a small application:

Boundary	Owns	Must not imply
Admission and queue	Accepted task identity, dispatch, delivery policy	Delivery means the task is still authorized
Durable state	Task lifecycle, attempts, ownership, budgets, effect ledger	A stored intention proves an effect happened
Executor and tool adapter	Bounded reasoning, artifacts, controlled effects	A successful tool response proves the entire task passed
Verifier	Evidence checks against the accepted task contract	A fluent completion message is sufficient evidence

Keep task state, attempt state, and effect state separate. A worker attempt can fail while its deployment succeeded. A task can be cancelled while an already-submitted effect remains unresolved. Flattening all three into a single failed flag loses the information needed for safe recovery.

Use ownership generations or another enforceable concurrency mechanism when replacing workers. An expired lease alone does not stop the old process from continuing. Stale owners must be rejected at the state and effect boundaries they could still reach. The same issue appears in multi-agent failure modes and final-write ownership, even when a diagram shows only one coordinator.

Scoped executors propose work and cross controlled effect boundaries

The executor loads a bounded working set, calls the model, invokes allowed tools, and records artifacts and evidence. It should not gain broad credentials merely because its prompt says it is responsible for the task.

Separate preparing a change from committing it. For the documentation example, an isolated workspace can hold proposed edits and build output. Publication passes through an adapter that checks the current task status, ownership, authorization scope, and exact artifact to publish.

That adapter needs a real concurrency contract. A status read followed later by an unconditional external call leaves a race with cancellation or reassignment. Define a durable dispatch commitment, reject stale owners before that commitment, and document what cancellation means after it. Once an external service has accepted an action, cancellation may require reconciliation or a separately authorized compensating action; it is not a promise that nothing happened.

Record the logical effect identity before dispatch, reuse it across retries where the provider supports idempotency, and retain the provider’s receipt or operation reference. After a timeout, classify the outcome as unknown until authoritative evidence resolves it. A local ledger entry saying “sending” is not proof of success or failure.

Different tools need different recovery strategies. Reading a document again is not the same operation as sending a customer notification again. The idempotency contract for agent actions explains why stable intent identity and reconciliation belong below the model’s retry decision.

Set separate limits for reasoning turns, tool calls, elapsed time, and external effects. Charge replacement attempts to the same task budget. Otherwise, each fresh worker can reset its local allowance while the overall task consumes unbounded resources.

Agent verifiers own evidence, not reassuring language

Completion should be a checked state transition. The executor proposes that the task is done and supplies evidence; the verifier checks that evidence against the accepted contract.

For a documentation publication, a build result checks one property of one artifact. A deployment receipt checks another. A live HTTP response checks availability, but a 200 response containing yesterday’s content does not prove the requested update shipped. Verification should connect the requested revision to the built artifact, the deployment reference, and a live content check appropriate to the task.

Bind evidence to identities: task, input revision, candidate artifact, execution attempt, verifier version, and observation time. If the candidate changes after verification, its earlier passing result should not silently approve the new candidate. Promote the verified artifact itself or reverify the changed artifact.

Use deterministic checks where the contract permits them. For semantic judgments, a model-based grader may help, but record its rubric and uncertainty. A second model reading the first model’s summary is not an independent observation of external state.

The verifier can be a module in the same process. The important separation is authority and evidence: executor prose must not be able to redefine acceptance criteria or overwrite a failed result. Likewise, verification should not quietly acquire write authority and become another uncontrolled executor.

Expose honest outcomes. Distinguish verified completion, incomplete work, rejected evidence, and unresolved effects. A task with an unknown external outcome needs reconciliation, not a confident success message or an automatic fresh attempt.

Agent backpressure belongs before the model call

A queue makes overload visible; it does not remove overload. Bound admission and concurrency according to the resources downstream work consumes. Model limits, build capacity, database connections, and external API quotas can become different bottlenecks.

Track queue age as well as queue length. A small queue can contain work that has already missed its useful deadline. Revalidate deadlines and authority before expensive execution, not only when the request first arrives.

Make cancellation and operator intervention part of the durable lifecycle. Stop admitting new effects once cancellation wins the defined dispatch boundary, retain evidence for effects already in flight, and report what remains unresolved. Poisoned tasks should reach a bounded quarantine or review path instead of cycling through retries forever.

Observe task outcomes rather than only model latency. Useful signals include time from acceptance to verified completion, time awaiting reconciliation, attempt count per task, duplicate-delivery handling, rejected stale owners, and tasks that exhausted their budgets. Those signals tell you which boundary needs attention without assuming the model is the bottleneck.

Test production agent architecture at the handoffs

Start with failure injection where one component believes another has taken responsibility. These are proposed acceptance cases, not results from a completed test suite:

Injected condition	Required observation
Task transaction commits; queue publish fails	Durable dispatch work remains recoverable without resubmitting the task
The same message is delivered twice	One logical task progresses; duplicate delivery does not duplicate its effects
A worker resumes after replacement	Its stale ownership cannot authorize a new state transition or effect
An external action succeeds; its response is lost	Recovery reconciles the original intent before any replay
An artifact changes after its build passes	The previous evidence cannot approve the changed artifact
Cancellation races with dispatch	The defined commitment boundary determines the outcome; unresolved effects stay visible
Model calls exhaust the task budget	New attempts cannot reset the allowance and continue indefinitely

For each case, inspect durable records and the actual effect target, not just the worker’s final text. A passing assertion about a log message is weaker than a check that the external system contains exactly the intended change.

The agent recovery checkpoint guide covers what must survive a restart. Apply the same discipline to every handoff: identify what has committed, what remains uncertain, and which component owns the next decision.

Start with the fewest deployable components your workload needs. A database-backed task runner with explicit contracts can be easier to reason about than a fleet of loosely coordinated agents. Split services when isolation, scaling, or ownership requires it, while preserving the same invariants.

A production architecture is not the collection of boxes around the model. It is the set of promises that still hold when a message repeats, a worker disappears, or a successful action loses its reply.