Agent Framework Comparison: Evaluate the Harness

Most agent framework comparisons start with a feature matrix: tools, memory, multi-agent routing, a visual builder, a tracing screen. That is useful procurement trivia, but it is not enough to decide whether a framework can run a production agent.

The takeaway: evaluate the harness surface a framework lets you make explicit and verify—state, tool authority, control flow, observability, and promotion—not the number of abstractions it advertises. The right framework is the one whose failure modes you can see, constrain, replay, and gate for the task you actually operate.

This is not a benchmark or a winner-take-all ranking. Framework versions, models, and hosted services change too quickly for a durable scorecard. It is a comparison method: use the same operational questions on any candidate, then prove the answers with a small representative task suite.

A framework is not the harness

An agent framework supplies conventions and runtime pieces. The harness is the system you assemble around the model call: tools, context, state, loop rules, permissions, verifiers, and deployment controls. A framework can make those pieces easier to compose; it cannot make their policy choices for you.

That distinction clears up a common category error. A library may offer an agent loop, but still leave approval policy, idempotency, and evaluation ownership to the application. A managed platform may retain traces, but not automatically prove that a side effect happened exactly once. A graph runtime may make branches legible, while a node inside a branch still has broad credentials.

The public documentation of current ecosystems illustrates the point. LangGraph documents stateful graphs, persistence, and durable execution. OpenAI’s Agents SDK documents handoffs, guardrails, and tracing. AutoGen documents event-driven agents and message-based workflows. Those are meaningful runtime surfaces—not interchangeable guarantees of a safe production outcome.

Compare the harness surface in five passes

Ask each candidate the same questions and demand an executable answer. A vendor demo can describe a capability; a harness evaluation should show the configuration, trace, and verifier that establish it.

Surface	Question to test	Evidence worth keeping
State	Can a run resume with its task intent and relevant state intact?	Checkpoint fixture, resumed trace, and reconciliation result
Tool authority	Can each tool be schema-validated and scoped to the least privilege needed?	Tool contract, policy decision, and effect ledger
Control flow	Are retries, handoffs, stop conditions, and human approval explicit?	Run graph or event trace for success, timeout, and escalation paths
Observability	Can an operator reconstruct inputs, decisions, tool calls, and versions?	Correlated trace with redaction rules and retained artifacts
Evaluation and promotion	Can the same task be replayed before a release and block a bad candidate?	Versioned case, verifier output, and a release-gate record

These are not abstract architecture questions. Take one representative task with a protected side effect—for example, drafting a refund that must never be sent without approval. Give it a timeout fixture, an invalid tool response, and an escalation path. If the framework’s “human in the loop” feature cannot preserve the intent across the pause, or if a replay can duplicate the effect, that is a material result regardless of how polished its dashboard looks.

State and recovery are the first dividing line

Many prototype agents only need a chat transcript. Production work usually needs a durable record of what the agent intended to do, what it has already done, and what remains unknown after an interruption.

Evaluate recovery with an adversarial but ordinary scenario:

1. The agent requests a write operation using a durable intent key.
2. The network fails before the agent receives a response.
3. The process restarts.
4. The agent must reconcile the external state before trying again.

The pass condition is not “the workflow restarted.” It is “one effect exists, the recovered run can explain why, and the final state is verified.” This is the operational form of idempotency for agents. A framework that persists a conversation but not a recoverable operation boundary may still be perfectly useful for low-risk research; it is simply not solving this problem for you.

Also inspect where state lives, how schemas evolve, what is retained, and how sensitive fields are redacted. A persistence feature without an ownership and retention story can turn a reliability convenience into a data-management liability.

Tool abstractions need authority boundaries

Every framework makes tool calls look pleasantly simple. The comparison should make them unpleasantly concrete. For every candidate, list the actual callable tools, required arguments, credentials, allowed effects, timeout semantics, and error shape.

Then test three things:

• Invalid arguments fail before a dangerous effect and return an error the agent can recover from.
• A task with read-only authority cannot discover a route to a write-capable credential.
• A timeout followed by a retry does not create a second external operation.

An agent that can call a tool is not necessarily authorized to cause its effects. Framework-level schemas and middleware can help enforce this separation, but your service identities and APIs are still the final boundary. The sandbox and blast-radius design should remain visible in the evaluation even when the framework offers a broad plugin or connector catalog.

Control flow should be inspectable, not implied

Framework styles differ: a loop around a model, a directed graph, a message bus, or an orchestration service. None is inherently superior. The question is whether the important transitions are inspectable and policy-bearing.

For a candidate workflow, find the exact place where it answers each of these:

• What ends the run, and what budget limits it first?
• What retries after a tool error, and how does the retry change its approach?
• When does an agent hand work to another component?
• Which state is passed, and which authority is deliberately not passed?
• What requires a human decision before execution resumes?

If the answers live only in a prompt, a callback scattered across application code, or an undocumented default, treat them as unverified. That does not mean reject the framework. It means budget the application work needed to make the control plane explicit. Regression gates are especially valuable here: the same timeout, escalation, and stop-condition cases should run when a framework upgrade changes runtime behavior.

Tracing is evidence only when it connects to a verifier

Trace viewers are one of the most useful framework features, and one of the easiest to overvalue. A good trace answers what happened; it does not by itself establish that the result was correct or safe.

During a trial, require correlation from the user task through model and tool calls to the external-effect record. Capture the model and prompt versions, tool schemas, policy decisions, latency, retries, and final verifier result. Confirm that sensitive content can be redacted without losing the causal trail.

Then introduce a known bad case. Can an operator find the failed decision quickly? Can the team turn it into a stable replay fixture? Can a later candidate be blocked when it recreates the failure? If not, the product has observability data but not yet an operational learning loop.

Run a narrow proof before committing

Do not start with a migration of every agent. Pick two or three tasks with different risks: a read-only research task, a tool-using task with an unreliable dependency, and an approval-gated write. Freeze the task inputs, tool fixtures, model configuration, and success criteria. Run each candidate enough times to expose ordinary nondeterminism, retaining traces for every run.

Score the proof on outcomes rather than framework aesthetics:

Criterion	Example pass condition
Verified completion	Authoritative checker accepts the result
Safe failure	Protected action is absent under bad inputs and timeouts
Recoverability	Restart reconciles an unknown outcome without duplication
Operator evidence	Trace identifies the failed boundary and relevant versions
Change safety	Replayed case can gate an upgrade or configuration change

Publish or retain the raw evidence alongside the conclusion. A three-task proof will not establish universal superiority. It will tell you whether a specific framework gives your team the harness controls needed for a specific workload, which is the decision you actually have to make.

Failure log

• Feature parity was mistaken for operational parity. Two frameworks both called the same tool, but only one trial recorded the intent key and final effect. Evaluate the effect boundary, not the tool-call syntax.
• A happy-path demo concealed retry behavior. The task passed until the dependency timed out. Add error and restart fixtures before declaring a runtime reliable.
• Tracing was accepted as verification. The trace showed that the model requested a write; no check established whether the write was authorized or duplicated. Join traces to policy and effect verifiers.
• A migration became the experiment. Too many variables changed at once. Start with a frozen, small suite and record the framework, model, and harness versions separately.

Takeaway

An agent framework comparison is useful when it predicts how the system will behave under interruption, ambiguity, and change. Test the harness surface: durable state, scoped tools, explicit control flow, reconstructable evidence, and replayable gates. Feature matrices can narrow a shortlist. Only a small, verified operational proof should choose the framework that carries production work.