Pentad Labs · Agent Operations · Context

The agent doesn't know what you know.

The failures practitioners blame on models are often failures of context. The agent did not know what the enterprise knows, could not find it in a usable form, or forgot what it learned last night. The model is rarely the defective part. The knowledge around it is.

An agent without institutional memory is a brilliant analyst with anterograde amnesia, hired daily and let go nightly.

A pattern runs through the reliability, cost, and operator-load problems in this series: something the platform should know, the platform doesn’t know. The context problem is that pattern at its widest and starkest. The agent doesn’t just lack a record of its own turns. It lacks the enterprise’s picture of the world and, most critically, of the enterprise itself.

An SRE asking about the operator’s version of this problem stated it plainly in a discussion of context rot:

Everyone knows their slice, but the full picture lives nowhere, so bit rot just keeps creeping in. — a practitioner, HN

The same complaint appears from the agent’s side, as the reason new memory systems get built at all. The builder of a purpose-built agent memory database described what he was replacing:

Memory layers that are either static vector stores or fragile prompt hacks. Retrieval is opaque, forgetting happens at the wrong time, and associations don’t form naturally. — a memory-system builder, HN

Enterprise knowledge exists in disconnected silos, scattered across databases and documents, owned by teams, updated on a schedule approximating randomness, and invisible to agents in the shape they could use.

The industry’s fix is retrieval over the raw estate

The industry’s answer to “the agent doesn’t know” is retrieval: point a search index or a text-to-SQL step at the existing data estate, and let the agent pull what it needs at runtime. The pattern is RAG and, by now, it’s quite mature for what it is.

RAG works for a reading task. It’s the wrong shape for an acting one. A retrieval step queries the enterprise one row at a time, in the vocabulary of the query, and returns whatever rows or passages the similarity surface thinks match. The agent then has the same problem the SRE described, scaled to machine speed: slices of the picture, retrieved one slice at a time, with no established relations between them. The builder quoted above is describing the consequence: retrieval that is opaque about why it returned what it returned, forgetting that happens at the wrong time, and associations that never form.

The deeper misfit is the divide the raw estate is built on. Enterprise knowledge lives in two forms that almost no system holds at once. Structured records, the rows in databases, with exact values and explicitly-defined relations; and documents, the prose where policy, procedure, business logic, and intent actually live.

In a good data shop, a database answers “how many units shipped to Toledo in Q3.” Only a document answers “what the Toledo customer was promised, and under which exception to which policy?” Retrieval over the raw estate returns one or the other. The agent acting in the enterprise needs both, joined.

This kind of knowledge has a shape

The reason “the full picture lives nowhere” is not that no one has written the parts down. It’s that the parts have never been assembled into the structure that relates them clearly. A customer is an entity across rows and tables and documents. A policy references procedures that reference roles that reference systems. The shape that expresses such relations is a knowledge graph; not a search index with a similarity cutoff, not a SQL join assembled ad hoc per query, but an explicit structure in which a thing and the things it is related to are first-class, navigable, machine-legible.

A platform that maintains this structure for the workload the agent is doing changes what “context” means. The agent does not retrieve slices and guess at the joins. It reads from a knowledge substrate in which the records and the documents are limbs of one memory, and in which the relations between them are already established. The answer to “what do we know about this customer” is a read against a maintained structure, not a fan-out of ad-hoc queries against the sources where the fragments live.

The ontology objection

A common impulse, on seeing that agents need structured context, is to reach for ontologies as the solution. Now the organization has two problems.

Explicit, machine-legible business logic is a good thing, and its goodness is exceeded only by its rarity. Ontologies work; they work at a cost of upkeep that has historically been prohibitive, because the writing of the ontology is the cheap part and the maintaining of it is the part that kills the program. Replacing the committee with a language model that writes the ontology halves the authoring cost, but does very little for the upkeep.

The way through is to notice what the ontology was for. Most of the questions an agent needs answered about business logic are not deep questions requiring a maintained formal theory. They are shallow questions: whether this customer is in scope for that policy, whether this action belongs to that role. They become answerable directly by an agent reading a knowledge substrate, provided the substrate is tight and current. The ontology exists to make answers legible to a machine. When the machine is an agent and the context is maintained, the legibility is a property of the context, which the substrate has assembled and maintains, rather than an artifact a committee has to produce. The hard questions that genuinely require the formal theory still exist. Those rare, corner-case exceptions are not where enterprise automation efforts go to die. They no longer justify the program.

This is the content of self-informing systems. When the word surfaced in the cost chapter, it meant a platform that gets cheaper because it learns from its own runs. The same property applied to knowledge is what the context problem is asking for: a system that builds and maintains its own picture of the domain it operates in, from the records and documents it governs. Now the picture the agent acts on is the same picture the enterprise could produce if you asked it the question directly. Not fetched per query; maintained continuously.

The memory is the same system as the record

The other four pages in this series argue that reliability, cost, operator load, and audit all fail because the platform lacks something. The context problem has the same shape, with one difference worth naming. The memory that tells an agent what the enterprise knows and the record that proves what the agent did are usually built as two separate systems by two separate vendors. That separation is a big reason why context rots. Value that only lives between systems is the most likely to rot first because entropy rules everything.

A platform in which the record of execution and the institutional memory are one system can maintain the picture as a side effect of governing the work. What the agent learned last Tuesday is in the same substrate as the proof of what it did last Tuesday. The same query language and even the same query answers both seamlessly. The SRE’s complaint, that context rots because the full picture lives nowhere, has a precise answer in a system shaped this way. The picture lives in the platform, because the platform is where the work happens.

An agent with that platform underneath it stops being a daily hire with amnesia. It becomes what the demo always implied it was: a colleague who knows what the organization knows.

What is an operating system for agents? defines the layer that provides memory as a service. The substrate that holds records, documents, and their relations as one institutional memory is part of the system design.

Agent operations → Back to the first note →