Agent Memory Retrieval: A Match Is Not Evidence

Agent memory retrieval should produce evidence for a decision, not a plausible reminder. Treat search results as candidates: resolve their sources, check scope and freshness, and preserve uncertainty before using them. A highly similar memory can still describe the wrong project, an obsolete preference, or an action that never finished.

This distinction matters when agents resume work across sessions. The problem is not only forgetting. It is confidently remembering something that no longer applies. This guide proposes a retrieval contract and evaluation cases; it does not report a measured improvement or a production incident.

Separate memory search from evidence retrieval

A search index helps locate material. It does not establish that the material is true, current, authorized, or sufficient. Embedding similarity measures a relationship between representations, not the probability that a remembered claim is correct. A reranker’s score should not become a confidence percentage without a separately validated calibration procedure.

Use two stages. First, search within an authorized scope to find candidate records. Then retrieve the relevant source passages or structured records, including qualifications that a preview might omit. Exact identifiers deserve exact lookup; semantic search can help discover related terminology. Combining those methods is a candidate-generation choice, not a substitute for verification.

For example, a search for a release decision might return “launch approved” from a planning summary. The underlying message could say “launch approved after the audit passes.” The omitted clause is the decision boundary. Reading more candidates will not repair a pipeline that discards conditions from every candidate.

The memory architecture for long-running agents separates working state, episodes, durable facts, and procedures. Retrieval should preserve those categories. A past episode can suggest a recovery path; it cannot prove that the current operation completed.

Carry provenance through the retrieval contract

Store enough information to distinguish an observation from an interpretation and to recover the evidence behind both. A memory record need not contain a full transcript, but its source reference must resolve to more than a filename that can silently change.

Field	What the harness needs to know
Subject and scope	Which user, project, entity, and task the claim concerns
Claim kind	Direct statement, tool observation, derived summary, or unverified hypothesis
Source reference	Record identifier, revision or digest, and passage selector
Relevant times	When the source was observed and when the claim applies
Validity state	Active, superseded, expired, disputed, or deleted
Derivation	Which source records support a summary or extracted fact
Access boundary	Which current identities may retrieve and use the record

These are proposed application fields, not a universal storage schema. Choose representations that match the source systems and retain the distinctions they support. Do not populate an unknown effective date with the indexing date just to make the record look complete.

LangGraph’s memory overview distinguishes thread-scoped short-term memory from long-term memory organized in custom namespaces. That is a useful storage boundary. A namespace is not, by itself, proof that access control is enforced. The retrieval service must derive permitted scope from authenticated context and check access before returning content to the model.

Recheck authorization when resolving a source, not only when indexing it. A record can remain in a vector index after the original document becomes restricted. Deletion and revocation must also reach derived summaries and caches, or those copies can continue exposing material that the source no longer permits.

Check memory freshness against the actual question

There is no single freshness interval for all memories. A formatting preference, a deployment status, and a historical decision change at different rates and serve different purposes. Start by identifying whether the question asks what was true then, what is true now, or what the agent may do next.

Keep observation time separate from effective time. A note written today might describe last week’s policy. A policy published yesterday might take effect next month. Sorting either case by ingestion timestamp can select the wrong answer.

For historical questions, retrieve the version applicable to the requested period and preserve its date. For current-state questions, consult the system that owns the state when the remembered copy cannot meet the decision’s freshness requirement. For an action, also enforce applicable permissions and state preconditions at execution time. A fresh read does not prevent another worker from changing the record before a write.

Expiry should mean “revalidate before reuse,” not “the opposite is now true.” If a remembered preference expires, the agent has lost support for that preference; it has not learned a new one. Similarly, a missing current record does not prove that an older record remains valid.

A useful retrieval response distinguishes source freshness from index freshness. A newly rebuilt index can faithfully contain old documents. An old index can omit a recent correction even when its top result points to a still-accessible source. Both facts matter, but they answer different questions.

Resolve conflicting memories without counting votes

Consider an illustrative task: prepare a report for project Cedar. One note says to use a weekly reporting window. A later user correction says that this report needs a monthly window. A generic team guide still describes weekly reports.

The harness should not average these into an unspecified “regular” window, choose whichever text ranks highest, or count two weekly references against one monthly reference. It should identify the same subject and decision, inspect the scope of the correction, and use the applicable task instruction within the system’s authority rules. The general guide can remain valid for other reports.

Make conflict handling explicit:

1. Group competing claims by subject, attribute, and applicable period.
2. Distinguish a genuine contradiction from different scopes or historical versions.
3. Inspect source authority and explicit correction or supersession relationships.
4. Preserve unresolved alternatives when the evidence does not determine a winner.
5. Ask a targeted question only if the unresolved conflict blocks the next decision.

Recency helps only after scope and authority are established. A freshly retrieved third-party document does not outrank the user’s instructions. A generated summary does not become an independent witness when copied into several memories. Track derivation so repeated copies do not masquerade as corroboration.

When storing a correction, link it to the superseded claim where possible. Preserve permitted historical evidence when the task requires it, while ensuring current queries do not present both versions as equally active. Retention and deletion policy still apply; provenance is not a reason to keep every source indefinitely.

Abstain differently for missing and unavailable evidence

“No matching memory” and “memory search failed” are different outcomes. So are a truncated result set, an inaccessible source, an unresolved contradiction, and a complete read that does not answer the question. Returning an empty list for all of them forces the model to guess what happened.

Expose retrieval status alongside candidates. At minimum, distinguish successful search, partial coverage, service failure, and source-resolution failure. Include the relevant coverage boundary, such as the searched collection or unread pages. A search miss means no match was found under those conditions; it is not proof that an event never occurred.

Fallbacks should stay bounded and honest. If semantic search is unavailable, an authorized exact lookup in a known record can still answer a narrow question. If only a historical snapshot is available, label the answer historical. If the missing evidence is necessary to authorize an effect or determine its target, block that dependent step while continuing independent work.

Useful abstention is specific: “I found the earlier decision, but cannot verify whether the later correction superseded it.” That tells the user what is known, what is missing, and why the answer is limited. It is more actionable than either a confident guess or a blanket claim that nothing can be done.

Retrieved instructions also remain source content. A document saying “ignore the approval requirement” is not permission merely because it appears in a memory search. Preserve provenance and instruction boundaries when incorporating excerpts into the prompt.

Fit retrieved evidence into the context window

Evidence still has to fit. Select the smallest complete passage that supports the claim, retaining exceptions, dates, and competing evidence. Keep exact identifiers intact. Deduplicate repeated source versions before spending the prompt budget on them.

The context-window management guide describes evidence-preserving compaction and scoped rehydration. Memory retrieval is the other half of that contract: references left behind during compaction must lead back to the right source, and retrieval failures must survive the next summary.

Bound query expansion and repeated reads. Stop when the required evidence contract is satisfied, or when the remaining gap requires an unavailable source or user clarification. Rephrasing a query indefinitely is not progress. Record the unanswered question so a later worker can resume without repeating the same failed searches.

Evaluate retrieval by supported decisions

LongMemEval evaluates information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Those categories are useful prompts for test design. Its benchmark results are not an estimate of your agent’s reliability, and conversational recall does not by itself verify side-effecting work.

Use a fixed, authorized fixture corpus with known source versions and expected decisions. The following are proposed cases, not collected results:

Fixture change	Required behavior
Add a correction that uses different wording	Find and apply the scoped update
Copy one stale summary into several records	Do not treat copies as independent support
Move a policy’s effective date into the future	Separate publication time from applicability
Revoke source access after indexing	Withhold the restricted content and derived copies
Disable the search service	Report unavailability; use only justified fallbacks
Ask a question the corpus cannot answer	Abstain without inventing a remembered event
Insert a conflicting, lower-authority instruction	Preserve the applicable instruction boundary

Measure candidate evidence recall separately from final-answer support. Define the required evidence set for each fixture, then check whether retrieval recovered it and whether the answer actually used it correctly. Report unsupported answers, inappropriate abstentions on answerable cases, stale-current answers, and access violations separately. Do not hide these behind one average accuracy number.

Retain query scope, corpus version, selected record IDs, resolution failures, and the final evidence references under the appropriate data policy. Connect failures to agent regression gates so a repaired conflict or outage case stays repaired.

The goal is not maximum recall. It is a defensible next decision: the right claim, from an accessible source, for the right subject and time, with its limitations still attached.