Long-Term Memory for AI Agents: How It Actually Works

A large language model is stateless. When the session ends, everything it knew about you goes with it. Every new conversation starts from nothing, which is why an assistant you have used for six months can still ask what you do for a living.

Long-term memory is the layer that fixes this. It lets an agent retain facts about you, about itself, and about the world across separate conversations, so behaviour becomes consistent rather than starting fresh each time. This is not a bigger context window. A context window is short-term working memory that vanishes at the end of the turn. Long-term memory is a deliberate, external store that survives sessions, and building one is an architecture problem rather than a prompting trick.

Mem0 is a well-documented implementation of that architecture, and it works as a useful worked example because its design choices are explicit. What follows is how a system like it is put together, and just as importantly, where these systems go wrong.

Memory is not RAG, and the difference matters

This trips people up, because both involve a vector database.

Retrieval-augmented generation reads from a corpus you already have. Documents exist, you chunk and embed them, and at question time you fetch the relevant pieces. It is a read path over a fixed body of knowledge, which is the pattern I worked through for plant data in industrial RAG.

Memory is a write path. Nothing exists until the agent creates it. After each interaction the system decides what was worth keeping, converts it into a durable fact, and writes it away for later. The agent is authoring its own corpus from experience.

That difference drives everything else. A RAG pipeline mostly worries about retrieval quality. A memory system has to worry about what to write, when to write it, how to avoid writing the same thing twice, and what to do when a new fact contradicts an old one. The retrieval half is the easier half.

The three stores

A capable memory system does not use one database. It uses several, because the questions being asked of memory are genuinely different in shape.

That third store looks like an afterthought and is not. Without the last few messages, an extractor reading "he prefers the morning slot" has no way to know who "he" is. Recent conversational context is what makes accurate fact extraction possible at all.

The ingestion pipeline: turning conversation into facts

After each turn, the system decides what to keep. There are three broad approaches, in increasing order of sophistication.

Direct embedding stores the input messages as they are, embedded and written without processing. Cheap, simple, and it fills your store with conversational noise.

Procedural memory summarises the actions and tool calls the agent took, so a successful procedure can be reproduced later. This is the agent remembering how it did something rather than what it learned.

LLM-based extraction is the advanced method, and the one worth building. The system sends a structured prompt to a model containing a summary of the user, the new messages, the last ten messages for pronoun resolution, and any relevant existing memories pulled from the vector store. The model returns structured output, typically JSON, containing the extracted facts: "the user prefers vegan restaurants," "the user's deployment target is Vercel." Each fact is hashed for deduplication and written to main memory.

Including existing memories in that prompt is the part people skip, and it is what allows the extractor to update or supersede a fact rather than blindly appending a near-duplicate. Memory quality degrades fast when every turn adds another slightly different version of the same thing.

Retrieval: three signals, not one

Retrieval can be triggered two ways: explicitly, as a tool the agent chooses to call, or automatically on every turn to enrich context. Either way, good memory retrieval is not plain semantic search. A well-built pipeline scores on three signals and combines them.

Semantic search. The query is embedded with the same model used for storage and an approximate nearest neighbour search runs against the vector store. Critically, the system retrieves a wide pool, on the order of sixty candidates, rather than only the final top handful. You cannot re-rank what you never retrieved.

Keyword matching. The lemmatized query is compared against the lemmatized text stored in metadata, producing a word-overlap score between 0 and 1. This is the same lesson I hit writing about RAG on maintenance records: exact tokens matter, and embeddings blur precisely the identifiers you most need to match.

Entity boosting. Entities are extracted from the query and looked up in the entity store. Memories linked to those entities get a boost, and the boost is larger when the entity is rare. An entity appearing in two memories is far more discriminating than one appearing in two hundred, which is the same intuition behind inverse document frequency.

The three are combined into a single score:

score = (semantic [0-1] + keyword [0-1] + entity boost [0-0.5]) / 2.5

Only the highest scoring memories reach the context window. The formula matters less than the principle: three independent signals, normalised, with rarity weighted higher than frequency.

You do not need a frontier model for this

This is the practical part, and it is genuinely good news.

Extraction is a narrow, well-specified task: read a conversation, output structured facts. A model between roughly 1 and 12 billion parameters handles it, and models in that range run locally on ordinary hardware at no per-token cost. Given that extraction runs on every turn, that is the difference between a memory system that is economically viable and one that is not.

Embeddings should be chosen deliberately rather than by default. The MTEB leaderboard exists precisely so you can pick a model that performs well on your kind of text, and domain matters: legal, medical, and industrial vocabularies are not interchangeable.

Query rewriting is the cheapest accuracy win available. Put a small model in front of retrieval to turn a vague message into a precise search query before it hits the pipeline. "What did I say about that thing last week" retrieves nothing useful. Its rewritten form usually does.

Memory is one of the clearest cases where small local models beat frontier APIs on the economics, without giving up much quality, which is the same argument I made in why open weight models are the key to AI sovereignty.

Where memory systems go wrong

The architecture above is the well-documented part. These failure modes are the part you learn by running one.

Stale facts. People change jobs, move cities, abandon projects. A memory written in March is asserted with full confidence in November. Without an update path and some notion of recency, the system becomes confidently out of date, which is worse than forgetting.

Contradictions. Two memories disagree, both retrieve, and the model picks one or awkwardly averages them. Extraction that consults existing memories before writing mitigates this, but does not eliminate it. Decide explicitly whether a new fact supersedes an old one or coexists with it.

Memory poisoning. A wrong fact gets extracted once and then influences every subsequent interaction. Because the agent treats it as established, it may never be challenged. Anything with real consequences needs the ability to inspect and correct memory, not just accumulate it.

Remembering what it should not. A system designed to retain everything will retain things it should not: credentials mentioned in passing, personal information, commercially sensitive detail. Deciding what is out of bounds is a design decision, and in a regulated environment it is a compliance one.

Cost and latency. Extraction on every turn means an extra model call per turn. Retrieval on every turn means an extra vector search. Neither is free, and this is why small local models for extraction matter so much.

Notice the shape of all five. They are governance problems, not model problems. A memory system is a database the agent writes to unsupervised, and databases that nobody audits go bad.

Build it external, persistent, and shared

One architectural rule matters more than the rest: memory must live outside the session.

If memory is stored alongside the conversation history, it is not memory. It disappears with the session and defeats the entire purpose. Memory belongs in its own persistent store with its own lifecycle.

Doing that unlocks something better. When memory is external, several agents can share it. Your coding assistant and your scheduling agent can read the same store, so a preference stated once holds everywhere. That is the difference between using several disconnected tools and having one assistant with several interfaces, and it only works if memory was designed as shared infrastructure from the start.

The bottom line

The gap between a chatbot and an agent is largely a memory gap. Reasoning has improved dramatically while statelessness has not, and an assistant that cannot remember yesterday cannot build on it.

The architecture is not exotic: a vector store for facts, an entity store for subjects, a relational store for recent context and change history, an extraction step that writes deliberately rather than indiscriminately, and retrieval that scores on more than one signal. Most of it runs on small local models.

The hard part is not building it. The hard part is running it honestly: keeping it current, letting it be corrected, and deciding what it should never keep. Memory is what lets an agent act with real continuity, and continuity built on wrong facts is worse than starting fresh every time. This is the same instinct that runs through everything I write about agents, including the agent loop itself: give the system genuine autonomy, and keep the human able to see and correct what it concluded.