Memory in the language the agent speaks

When people ask how to give an agent memory, the usual answer is a vector database. Embed everything, search by similarity, hope the right piece comes back. I reached for that too, for a while. I have come to think it is a holdover from before agents existed.

Back then, the way you gave a model anything beyond its training was to fit it into the single call you got. A system prompt, some retrieved chunks, and that was your shot. Embeddings were a reasonable way to choose the chunks. It made sense for the tools we had.

Agents changed the tools we have. An agent can look around. It can read a file, follow a path, run a search, and decide what to open next. Once that is true, the question stops being how to retrieve the right chunk. It becomes where to keep things so the agent can go and find them on its own.

Two kinds of memory

It helps me to split memory in two.

There is the memory that is always there, loaded in before the agent does anything. The small, important things it should never have to go looking for. Who the person is. What you are working on. The few facts that shape everything else.

And there is the memory it goes and fetches. The larger pile it does not need until it suddenly does. Old work, past decisions, a note from three weeks ago. That part should not sit in context. It should sit somewhere the agent can search when the moment calls for it.

In Akarii it settled into something plain. The searchable layer is just markdown files the agent greps and reads. The always-there layer is a compact index of what exists, plus a rolling weekly summary, enough to keep it oriented without pulling everything into context. The files are what it opens when it needs the detail.

Put it where the agent already lives

For the part it searches, the reflex is a vector store. I think the better home is a filesystem.

Agents are already very good with files. They read them, write them, grep them, move around a directory the way you would. It is the environment they are strongest in. Memory kept as named files in folders is memory in a form the agent can actually work with, instead of a black box it queries and hopes.

It fits how the agent reasons, too. A file has a name, a place, a date. "What did we do last week" is a question a filesystem can answer, because last week is a folder, or a date, something to navigate to. Semantic search cannot really do that on its own. It ranks by meaning, not by time. You can attach timestamps to your vectors and filter on them, but now you are writing the deterministic scaffolding the filesystem would have given you for free.

The wrong layer

There is a deeper reason the vector reflex feels off to me. Embeddings are how the model represents meaning inside itself. They are closer to neurons than to notes.

It is tempting to think that because the model runs on embeddings, its memory should too. But the agent does not read and write in embedding space. It reads and writes in language. The retrieval happens outside the model, out where the agent works, and out there the native medium is text. Reaching for embeddings to hold memory borrows the model’s inner machinery and uses it at the wrong layer.

Give the agent memory in the language it actually speaks, and let it use the tools it is already good at to go and find what it needs.

It comes down to a bet I have grown more sure of. Trust what the model is genuinely good at, reading, reasoning, noticing patterns, rather than wrapping it in machinery that decides for it. Memory is one more place it holds.

I would not be too absolute about it

Even in Akarii the agent kept a semantic search it could reach for, for the times the wording of a question did not match how the note was written. It just was not the front door. The filesystem was. And if you truly cannot give an agent files, a vector store on its own is a fine workaround. For very large piles of data I have not gone looking for the honest limits of this yet. Retrieval has its place.

But as a default, I would rather hand the agent an environment it understands and let it go looking, the way you open a drawer you already know the shape of. That is where the better tools seem to be heading. It feels like the right direction to me.