Agent Observability: What to Log and What to Ignore

The first thing most teams log about an LLM agent run is everything: full prompts, full completions, every intermediate thought. Then they go looking for why a run failed and drown in tokens. The second thing they learn is to log less. Agent observability is not about recording everything — it is about recording the six things you will actually search for when something goes wrong, and throwing away the rest.

What you actually need

A runnable observability layer for an agent harness captures three categories: tool calls, decision boundaries, and harness events. Everything else is either derivable from these or not worth the storage.

Tool calls — the ground truth

Tool calls are the only thing the agent does in the world. A prompt is a thought; a tool call is an action. Log each one as a structured record:

{
  "step": 7,
  "tool": "search_web",
  "input": {"query": "cloudflare pages deploy command"},
  "output_summary": "3 results, 1420 chars",
  "duration_ms": 830,
  "error": null
}

Notice what is missing: the full output. Most tool outputs are large and rarely useful in their raw form. Store a summary — character count, result count, status code — and write the full payload to cold storage keyed by run ID. When you need the full output for debugging, fetch it. When you don’t, it is not cluttering your dashboard.

The fields that matter on every tool call:

• Step index — where in the agentic loop this happened
• Tool name — which capability was invoked
• Input — the arguments, in full (these are small and high-value)
• Output summary — not the full payload
• Duration — wall-clock time, because slow tools are slow tools
• Error — null or the failure message, not a stack trace

Decision boundaries — where the model chose

A decision boundary is any point where the agent chose between paths: called tool A instead of tool B, decided the task was done, chose to retry after an error. You do not need the model’s reasoning transcript. You need a one-line summary of the fork:

step 7: chose "search_web" over "read_file" (model citation: "need external docs")
step 12: decided task complete (verifier passed, 0 retries remaining)
step 4: retried after tool error (attempt 2/3)

This is the layer that turns a post-mortem from archaeology into a five-minute lookup. When a run fails, you trace the decision boundaries backwards: where did the agent commit to the path that led to the failure? That step is your culprit, and the one-line summary tells you why the model thought it was right at the time.

Harness events — what the scaffolding did

Guardrails firing, budgets hitting their ceiling, kill switches tripping, retry policies activating — these are harness events, not model decisions. They are the rarest log entries and the most important:

[BUDGET] token budget exhausted at step 14 (8192/8192)
[CIRCUIT_BREAKER] repetition detected: search_web called 3x with identical args, steps 11-13
[KILL_SWITCH] operator interrupt at step 9

Harness events are how you tune the guardrail layer. If your repetition breaker trips on a legitimate retry (the tool was flaky, not the model stuck), you adjust the threshold. Without the event log, you are tuning blind.

What to throw away

Most of what teams log on the first try is clutter. Here is what to cut:

Full prompt text on every turn. The prompt is reconstructible from the tool call log plus the system prompt (which is fixed). Logging it every turn doubles your storage for zero insight. Log the system prompt version hash, not the text.

Full completions. The completion is the model’s internal reasoning. Unless you are debugging a prompt change (use versioned prompts for that), the completion is derivable from the inputs and the model version. Log the model ID and a hash, skip the body.

Successful runs at full fidelity. Sample them. Keep 1-in-10 success traces at full resolution; keep the rest at summary level. You will look at failures fifty times more often than successes.

Context window contents. If you need to know what was in the window at step N, you can reconstruct it from the tool call log and the initial context. Logging the full window every step is O(n²) storage for O(1) insight.

The pipeline: logs become evals become guardrails

Observability is not a destination — it is the input to the rest of the harness. The sequence:

1. Logs accumulate from production runs.
2. Failure traces (the subset where a harness event fired or the output was wrong) become eval cases. Each failure trace is a regression test: this is what went wrong, this is the task, this is what the model did. Add it to your eval suite.
3. Patterns in failures become guardrails. If you see the same repetition loop five times across different runs, that is a circuit breaker waiting to be born.

The teams that build good agents fast are the ones whose observability layer feeds their eval layer feeds their guardrail layer. Each production failure makes the next run safer. Without observability, failures are stories; with it, they are data.

The cost question

Observability is not free. The overhead of structured logging, even at summary level, adds latency and storage. The practical approach:

• Log synchronously for failure paths (you want the trace when things go wrong)
• Log asynchronously for success paths (buffer and flush)
• Set a storage ceiling — 90 days of traces, then archive or delete
• Use structured JSON, not free text. Every field you cannot query is a field you should not store

The rule of thumb: if your observability layer adds more than 5% to run latency, you are logging too much. Cut the success traces harder, keep the failure traces full.

Takeaway

Record tool calls, decision boundaries, and harness events. Throw away full prompts, full completions, and full context dumps. Feed your failure traces into your eval suite and your failure patterns into your guardrails. The question is never “do we have enough logs” — it is “can we find the cause of the last failure in under five minutes.” If yes, the observability layer is doing its job.

For how to retain the outcomes these traces support without turning the prompt into an archive, see Memory Architectures for Long-Running Agents.