Pentad Labs · Reference · The Memory
An agent's memory belongs to the enterprise. An agent OS makes it so.
An AI agent memory is what an agent learned and recalls across turns and runs: the facts it was given or extracted, where each came from, and how far to trust them. When memory is managed by the agent OS rather than by the agent or its harness, it can be audited, corrected, and governed.
The agent's account of what it knows is testimony. But a memory held by the agent OS is evidence. They are not the same thing, and only the memory the agent OS holds can answer what was known, when it was known, and on whose word at any arbitrary future time, that is, under threat of audit.
Agents don’t live for one turn. They run a session, get picked up again next week, hand work to other agents, and need to, but can’t on their own, answer for decisions made months ago. Something has to persist what they saw and did. The market calls it persistent memory, long-term memory, or a memory layer, and sells it as a standalone feature: better recall, personalization, smarter agents.
The questions this page asks, and the answers it gives, are different: who holds the memory, who can correct it, and who answers for it when someone asks later on?
The enterprise’s challenge
At some point an agent acted on a fact neither you nor it can produce today:
- An exposure figure the credit desk revised in August.
- A policy the partner stated in June and changed in September.
- A customer’s standing the compliance unit had held differently all along.
Probably the decisions were defensible given what the agent knew then. The questions that reach you six months later are a lot harder to answer: what it knew, where that information came from, and how far it should have been trusted.
The agent can’t answer these and a better model won’t answer them either. The context window it reasoned in is gone. If it kept a private store, what is in the store is its account of what it knew, written by the same process that made the decision you are now asking about. Memory that answers to the decision an agent produced is testimony. Testimony from an interested party isn’t evidence. Testimony is what the interested party says. But evidence is what you can check without taking the interested party’s word for it.
The questions memory exists to answer, what was known and when it was known and on whose word, can’t be answered by the agent about itself.
So there are two problems: answering these questions after the fact and doing so in a way that agents can’t game. The solution to both of these is to store memory where the agent can’t game it. Doing that achieves four benefits in the enterprise:
- Corrections land once. Fix a fact and every future read reflects it, whatever agent makes the read, instead of one private silo per agent, each with its own wrong copy.
- Answers for the auditor. What was known and when it was known becomes answerable at all, because the answer comes from the record rather than from the agent’s account of itself.
- Partners you can say yes to. A partner’s agent can work on data you control, because what it carries home is the data owner’s decision rather than the visitor’s. That’s the difference between saying yes to agents and leaking to them.
- Learning accrues to the enterprise. A record that outlives the agent accrues value based on what the enterprise has learned, regardless of whose model did the learning.
This page is a companion of the agent audit trail. An audit trail is a record of what the agent did, kept outside the agent so its actions can be reconstructed and replayed. Memory is, by contrast, the record of what it knew: the facts it found, their lineage, their standing, their trust.
Both of these records belong outside the agent, for the same reason.
That’s one of seven issues with agentic memory as conventionally implemented in an enterprise setting. We’ll discuss these in turn below, but here they are in one place.
| Enterprise needs | What an agent does | What the agent OS does instead |
|---|---|---|
| Persistent memory: the agent remembers between sessions | Loses the thread inside a session; compaction has a floor no method beats | Assembles the context block every turn from the append-only record; the working set is a view rebuilt from it |
| Facts stored as they actually happened, without hallucinations compounding | Writes what its extractor emits, hallucinations included, and yesterday’s error becomes tomorrow’s context | Extraction before ontology; the write path is the substrate’s, not the agent’s |
| Outdated facts corrected once, everywhere | Cites a stale fact with full confidence, because it can’t know it’s stale | Append-only, decay and retraction; every hit arrives with lineage and a currency signal |
| Conflicting records kept as conflicting, not flattened into one wrong answer | Reads one reconciled value and can’t tell what was set aside to make it | Keeps the perspectives, fuses them only per decision, records what the fusion set aside |
| Every fact traceable to who said it, and how far to trust them | Offers only its own account of what it was told | Opinions with their own uncertainty, trust as a rating, a case file that doesn’t collapse |
| Shared memory between agents, so they work together without lockstep | Coordinates by messaging that fails on facts neither agent can state exactly | Shared memory with approximate match; coordination and memory are the same problem |
| Governance: a partner’s agent on your data, erasure on demand, answers for the auditor | Keeps what it wants and carries it home, ungoverned | Memory as a service of the agent OS; the take-home curated; the record kept apart from the agent’s own report |
A context window is a buffer, not a memory
You’ve seen an agent lose the thread inside a session and been told the fix is a bigger context window, which means a more expensive inference bill, or a summarizer that condenses memories before they roll out of the context window. Two things are often mistaken for agent memory: the context window and the vector store. A context window is a buffer. A vector store is an index. Neither is a memory your organization can answer for.
The context window is bounded per-turn, so typically the harness decides what stays in it. The usual answer is compaction: a summarizer condenses what is about to fall off the edge. Compaction has limits, which are proven in Context Compaction Theory, including a lower bound no method beats no matter how it computes. Measured against that floor, Anthropic’s Opus 4.8 compaction endpoint, at a budget of roughly fourteen kilobits over fifteen thousand items, can’t tell members from non-members: its total error sits on the random-guess line. Wunderblock, the vector-symbolic memory substrate in WunderOS, run at the same task shape and budget, lands at 0.245 to 0.310 total error, 84 percent of the way to the floor. A Bloom filter at the same budget manages 55 percent. Managing the buffer well is substrate work. The receipts
The buffer was never the agent’s own work product. In WunderOS a component called the virtual memory window assembles the context block for every turn: a fixed token budget divided into named slots, each with an owner, a priority, and an eviction policy.
The working-memory slot isn’t a store. It’s a materialized view, rebuilt at session start by replaying the append-only record of episodes, and if it’s corrupted the recovery path is to rebuild it from the record, which is also the normal startup path.
What the agent holds in front of it is derived state. The record it was derived from is the memory, and the record is the substrate’s.
What gets remembered is decided when it’s written
What gets written is decided at ingestion, when text is turned into facts, and if you didn’t design that decision it was made by whatever you pointed at the text. The tempting design fixes a small set of relation labels and makes extraction the act of sorting each sentence into one of them. “Treasury terminated the counterparty’s credit line in March” states a termination. If the label set contains no termination, the extractor either forces the sentence into a label that distorts it or records nothing, and the thing the text said is gone before any query has had a chance to want it.
Extraction should recover the predicate the text states, in the text’s own vocabulary, and the mapping to the system’s preferred forms should be a separate, later, reversible stage. Extraction before ontology
Then there’s rot. Agent output is adversarial to memory quality. A model that hallucinates in session N writes its hallucinations somewhere, and any memory that persists them feeds them back as session N+1’s context, where they ground more of the same. That’s the expected failure mode of ungated persistence. The defenses are substrate functions applied at write time and at materialization: quality gating against what is already known, contradiction detection against what came later, lineage confidence carried on every write.
The writer of the garbage can’t be the curator of the garbage, so the write path is the substrate’s, and the agent’s own output reaches memory through a gate the agent doesn’t control.
Memory decays and retracts; it never deletes
If you maintain a store, never delete sounds like malpractice. It’s the right default anyway, and the argument is one you can check against your own requirements. Three properties argue against destructive deletion:
- Audit. A store that never deletes can answer what was known and when it was known, for any past instant, because the record of every assertion and every retraction is still present. A store that deletes can’t.
- Isolation. Deletion is a write that mutates shared structure, and mutation under concurrency across tenants is where isolation guarantees are hardest to keep. Append-only confines each tenant to additions.
- Replay. A destructive delete is an event whose effect depends on what was present when it ran, so replaying it against a divergent prefix produces a divergent result. A retraction tombstone is additive and replays the same regardless of order. Why append-only
The lifecycle operations everyone asks for are realizable at retrieval time:
- Redaction is a tombstone.
- Retirement is decay of retrieval weight until the fact stops clearing the threshold.
- Supersession is a typed edge the retrieval layer reads.
- Relaxation is per-fact authority weighting.
Staleness is a problem of how much authority a fact carries when it is retrieved, not a storage problem, and solving it by erasure spends audit and replay to fix something scoring already fixes. A retraction has to bind every future reader at once, and only the store every read passes through can do that.
Logical retraction doesn’t satisfy legal erasure. A GDPR right-to-erasure request requires that named records be physically removed, and a tombstone leaves the content in place. Meeting it takes a destructive escape hatch that breaks the append-only invariant deliberately, with the break itself an audited event: the store can answer what was erased, by whom, under what authority, and when, though it can no longer answer what the erased records contained. Retraction is the mechanism. Erasure is the named exception.
Retrieval returns authority, not just relevance
Every hit your retrieval returns makes a claim about how far it deserves to be believed, and most systems make the claim silently. A correctly retrieved stale fact misleads when it arrives as a bare match, so the consuming agent thinks of a low-authority fact as a full-authority one. A hit should arrive carrying its lineage and a currency signal, so the consumer receives not only that a fact matched but how current it is and on what trust. Relevance is what an index returns. Authority is what a record carries, and only the party that holds provenance can attach it.
The same requirement holds one level up, where the enterprise has no single true record of anything. The customer the credit desk knows, the customer the marketing line knows, and the customer the compliance unit knows are three partial views, held with their own evidence and their own trust, and they do not agree.
The conventional answer is the golden record: elect one canonical value and oblige every line of business to reconcile to it.
That buys consistency by discarding the disagreement, and the disagreement was information. The memory keeps the perspectives, fuses them only when a decision forces a single frame, and records what the fusion set aside. No view from nowhere
Memory has to keep who said what
If you run several agents, two of them already disagree about something, and the only account either can offer is its own. The standard taxonomy of agent memory, taken over from cognitive psychology, is working, episodic, procedural, semantic. It’s a good taxonomy. It also assumes two things silently: that memory belongs to an individual, and that agent memory should be modeled on human memory at all. An enterprise isn’t an individual. It’s a society of observers, and its memory has to keep what each observer said, from which vantage, with what confidence, and how far the others trusted it.
Four pieces do that work.
The opinion. A belief is carried as an opinion with three masses, belief, disbelief, and uncertainty, rather than as a scalar, because a scalar can’t tell a source examined and found evenly balanced from a source about which nothing is yet known.
Fusion. Opinions fuse by addition of evidence, which doesn’t depend on the order of arrival, so two agents that have seen the same accounts hold the same combined view.
Trust. Trust in a source is a rating with an uncertainty of its own, updated by outcomes and auditable like any other fact.
The case file. The views of one matter live together in a file that doesn’t collapse them. The social tier
Who said what is a fact about the fleet, not about any one agent, so it lives in the tier the fleet shares.
Shared memory is how agents coordinate
The coordination you’re probably building is messaging: queues, topics, request and reply. It works until two agents have to share a fact neither can state exactly, and then it doesn’t. Coordination and memory are the same problem. Linda’s tuplespace, the classic coordination model, gave distributed systems a shared associative memory accessed by exact match, and it never scaled past small systems, because exact unification is the wrong primitive for communication that is partial, noisy, or analogical: a template that doesn’t quite match returns nothing. Vector-symbolic architectures supply the approximate match. A tuplespace realized over one writes by bundling, reads by factoring against a template up to a caller-supplied threshold, and gets wildcards, analogical queries, and salience that grows with use as consequences of the algebra rather than features added to it. A coordination space and a memory space are the same data structure with different parameters, which is an implementation fact rather than a metaphor. Vectorspaces
An agent that owns its memory can coordinate nothing beyond itself, because coordination is communication through a shared store and the approximate matching over it is the mechanism.
The agent OS governs the memory, so you can get work done
Everything about an agent can be governed at the boundary around it. What it may touch, what it is shown, what it may do, what gets replayed, what gets unwound: each is a property of a boundary.
The one thing a boundary can’t reach is what the agent keeps: memory inside the agent is inside the box, and a boundary can’t reach inside a box it has agreed not to open.
Governing what a visiting agent remembers when it leaves is reachable only if the memory isn’t inside the agent.
That’s the architectural commitment: memory is a service of the agentic operating system, not a possession of the agent. When the memory of a visit belongs to the host, curating it is an ordinary operation on the owner’s own store rather than a negotiation with the visitor, and an unmodified agent and a fully governed agent become the same fact. The agentic data enclave
The difference between this and a harness is what the word syscall does, and it’s the line to check when you evaluate a platform: does the governance run on what the agent volunteers, or on what the agent can’t avoid? A harness governs what the agent volunteers to route through it. An agent OS runs the agent as a guest, and every memory write, every tool call, and every byte that would leave is a syscall the kernel mediates whether the agent consents or not. The agent arrives with no memory of its own, works inside the boundary, and leaves with exactly what the boundary passed and nothing it didn’t.
The closest rival claim runs the metaphor the other way. MemGPT proposed the model itself as the operating system, managing its own memory hierarchy, context as RAM, store as disk. The roles are backwards. An operating system isn’t the thing that holds state. It’s the thing that answers for state, to someone other than the process. A model that pages its own memory is a process with a cache, and its report on its own state is the only account there is, which is testimony again.
Replay the run from the audit trail, and at every step you can ask what the agent knew: which facts were in front of it, on whose word, with what standing. The trail answers for what it did. Memory answers for what it knew. Both are held by the agent OS, and that’s what makes them answerable. The agentic data enclave is where this becomes urgent, and what is an operating system for agents is the platform that holds the services.
A FAQ about agent memory
Why does my agent forget everything between sessions? Because the context window is a buffer, not a store, and whatever lived only in it is gone when the session ends. The usual fix, a summarizer that compacts what fell off the edge, has a measured floor it can’t get under. The fix that holds is the other way around: end each session by writing what was learned to an append-only record, and rebuild the next session’s working set from it. The agent doesn’t remember; the enterprise does, and hands it back every session.
Is RAG agent memory? No. RAG is a read primitive. It moves text into the context window at retrieval time and creates nothing a later turn can answer from: no record of what was retrieved, no lineage on the hit, no currency, no retraction. A memory system can use retrieval as one of its mechanisms. The memory is what the substrate keeps, with the authority attached.
Is a vector database agent memory? An index returns similarity. A vector store can tell you what is near the probe. It can’t tell you who said a fact, when it stopped being true, or that it was retracted last week, because similarity has no notion of any of those. It’s a place to put bytes and fetch them back. What makes memory governable isn’t the embedding. It’s the record the facts live in and the authority that retrieval attaches to them.
Can an agent forget, and how does that square with GDPR erasure? It shouldn’t delete. It should retract. Decay lowers a fact’s retrieval weight until it stops clearing the threshold, and a tombstone excludes a wrong fact from results while keeping its lineage. Neither satisfies a GDPR Article 17 request, because the content is still physically present. Erasure takes a destructive escape hatch that breaks the append-only invariant deliberately and records the break.
How do multiple agents share memory? Through a shared substrate with approximate matching as the primitive. Each agent holds a working view materialized from the shared record, what agents commit lands in shared tiers under consolidation, and an agent knows what another did when that fact is in its lineage. Coordination and memory are the same problem, so shared memory is the coordination mechanism, and exact-match messaging is the wrong primitive for agents whose words don’t line up exactly.
Does agent memory need an LLM to work? No, and putting one on the path makes it worse at the job as measured. WunderOS runs no LLM on the memory hot path: extraction is a deterministic dependency parse, storage is vector-symbolic bundling, and the context block is assembled by a fixed-budget slot protocol. Against the information-theoretic compaction floor at a matched budget, the LLM summarizer sits on the random-guess line and the deterministic substrate is 84 percent of the way to the floor. An LLM in the memory path is a lossy summarizer with a floor.
What’s the difference between agent memory and an agent audit trail? The audit trail is the record of what the agent did. Memory is the record of what it knew. They are held outside the agent for the same reason, and they answer different questions: the trail reconstructs the run, memory reconstructs the knowing. Together they let you replay a decision and show what the agent saw when it made it.
Should an agent manage its own memory? The proposal has been made seriously, in the MemGPT line, and it fails on who answers for the record. An agent’s self-managed memory is its account of itself. The account reads the same whether it’s false or true, and its errors are the agent’s own output, which is the one input memory most needs gated. A memory the agent OS holds is checked rather than believed.