Memory in the language the agent speaks
I reached first for a vector database when building agent memory. In practice, plain files gave Akarii a more useful default, with semantic search kept as a supporting tool.
My first instinct for agent memory was the familiar one: put everything in a vector database, search by similarity, and feed the closest chunks back to the model. I used that approach for a while. Building Akarii made me question whether it should be the default.
For a single model call, retrieval works well. You have one prompt and a limited context window, so embeddings offer a reasonable way to choose which pieces to include.
An agent has more options. It can inspect a directory, read a file, run a search, and decide what to open next. That shifts the design question from “which chunk should we inject?” to “where can the agent reliably find this later?”
Two kinds of memory
I find it useful to separate memory into two layers.
The first is always present: a small set of facts the agent shouldn’t have to search for, such as who the person is, what they’re working on, and the preferences that affect most tasks.
The second is fetched when needed: old work, past decisions, and notes that may matter once in a while. Loading all of that into every context wastes space. It belongs somewhere searchable.
Akarii ended up with a deliberately plain setup. The searchable layer is a collection of Markdown files the agent can grep and read. A compact index and rolling weekly summary stay in context to keep it oriented. When it needs detail, it opens the relevant file.
Put it where the agent already lives
For Akarii’s searchable memory, the filesystem proved to be a better front door than the vector store.
Agents already know how to work with files. They can read, write, grep, rename, and navigate directories. Named files also expose the structure of the memory instead of hiding it behind a similarity query.
Files come with useful clues: names, locations, and dates. For a question such as “What did we do last week?”, time may matter more than semantic similarity. You can add timestamps and filters to a vector store, but the filesystem already provides much of that structure.
The wrong layer
There’s also a mismatch in how the memory is represented. Embeddings are a useful machine representation of meaning, but they aren’t how the agent reads or writes its notes.
At the tool layer, the agent works in language. Text files let it inspect, edit, and reorganise its own memory with the same tools it uses elsewhere. A vector database is excellent at retrieval, but it is less transparent as a working environment.
Keeping the source memory as text lets the agent find and maintain it with tools it already understands.
This reflects a broader preference I’ve developed while building agents: give the model legible material and let it use its reading and reasoning abilities before adding machinery that makes every retrieval decision in advance.
I would not be too absolute about it
Akarii still kept semantic search for cases where a question used very different wording from the original note. It just wasn’t the first option. A vector store is also a sensible choice when the agent can’t access files. And I haven’t tested the filesystem approach against every scale of dataset; retrieval clearly still has a place.
For the products I’ve built so far, I’d start with files and add semantic search where it earns its keep. The setup is simple, inspectable, and surprisingly capable.