5

Mins

The Memory Layer: Why Your AI Agent Forgets and How to Fix It

AI models forget everything between chats. Here's how to give your agent a memory, and keep it safe.

memory-layer

CONTENT

Key Takeaways

  • AI agents need short-term and long-term memory to retain context, user preferences, past events, and task instructions across conversations.

  • Mem0 and LangGraph offer practical ways to store, retrieve, update, and manage agent memories.

  • Secure memory systems require source tracking, user isolation, expiry dates, deletion controls, and protection against memory poisoning.

Why agents need memory

Tell a support chatbot about a double charge on Monday. Come back on Tuesday and ask about the refund. It asks for your order number again. The bot isn't careless. It never knew you.

Every call starts from zero

An AI agent is a large language model (LLM) that uses tools to finish a task. ChatGPT and Claude are built on LLMs. The LLM itself is stateless. That means it keeps nothing from one request to the next. OpenAI's docs say it plainly: "each text generation request is independent and stateless."

So how does a chatbot remember your last message? The earlier messages are sent back to the model with every new request. Close the chat, and that history is gone.

What breaks without memory

  • The agent asks the same questions every time.

  • Work on a long task is lost when the chat ends.

  • Every user gets the same answers.

Short-term vs long-term memory

  • Short-term memory is the context window: all the text the model can read in one request. Think of it as the model's desk. It's small, and it's cleared after each chat.

  • Long-term memory is a separate store, like a database, that keeps facts between chats. Think of it as a filing cabinet next to the desk.

Memory is not the same as state

  • State is the agent's to-do note for the current task: which step it's on, and what its last tool returned.

  • Memory is what's worth keeping after the task ends.

two-chats-without-long-term-memory

Figure 1. Two chats, with and without long-term memory. Conceptual Example.

Words you'll see in this article

Term

What it means

Token

A small piece of text, often part of a word. Models measure text in tokens.

Prompt

The text sent to the model. The system prompt holds its standing instructions.

Tool

Something the agent can use, like a web search. What comes back is a tool result.

Framework

Ready-made code for building agents, so you don't start from scratch.

The four types of memory

In plain words: The 2023 CoALA paper (Cognitive Architectures for Language Agents) uses all four. LangChain's docs use three: semantic, episodic and procedural. They call the chat short-term memory.

  • Working memory: what the agent holds right now, usually the context window.

  • Episodic memory: past events. Example: "Last month, this customer was charged twice and got a refund."

  • Semantic memory: facts. Example: the customer is on the Pro plan and prefers email.

  • Procedural memory: how to do things. Example: "Always check the order number before a refund." It lives in prompts, code and the model.

๐Ÿ’ก Good to know: Some agents rewrite their own instructions from feedback, updating their procedural memory.

the-four-memory-types

Figure 2. The four memory types for one support agent. Examples are illustrative.

Knowledge vs memory

In plain words: Both end up in the context window, but they're different things.

  • Knowledge is shared. Your team writes and checks it: help articles, policies, product guides.

  • Memory is personal. The agent writes it as it works: this customer's plan, what they asked last week.

RAG: how knowledge gets in

RAG stands for retrieval-augmented generation. It's like an open-book exam: the agent looks up the right pages before it answers. The name comes from a 2020 research paper. It works in three steps:

  1. Chunking: cut each document into short passages.

  2. Embedding: turn each passage into a list of numbers that captures its meaning. Passages with similar meanings get similar numbers.

  3. Retrieval: when a question comes in, find the closest passages and add them to the prompt.

Why they need different rules

  • Knowledge changes only when your team edits a document.

  • Memory changes in every chat. It belongs to one person, and it needs an expiry date and a delete button.

๐Ÿ’ก Good to know: Keep them apart. Otherwise, one customer's details can show up in another customer's answer.

shared-vs-personal-memory

Figure 3. Shared knowledge vs personal memory.

Managing the context window

In plain words: The context window fills up as a chat goes on. A bigger window doesn't fully solve this. Anthropic's engineers describe "context rot": the more text in the window, the worse the model gets at recalling it. Three habits help:

  • Summarise old messages. When the window is nearly full, swap the old chat for a short summary. Anthropic calls this compaction.

  • Clear old tool results. If the agent read a file 40 steps ago, it can read it again when needed. Anthropic calls this "one of the safest lightest touch forms of compaction."

  • Save key facts to long-term memory. Then load back only the facts the current task needs.

full-chat-vs-summarising

Figure 4. Full chat vs summarising.

What Anthropic measured

On September 29, 2025, Anthropic added two features to the Claude Developer Platform, in public beta.

  • Context editing clears old tool results automatically.

  • The memory tool lets Claude save files outside the context window.

Anthropic tested them on internal search tasks, but doesn't detail its baseline.

  • Context editing alone improved performance by 29%.

  • Context editing plus the memory tool improved it by 39%.

  • In a 100-turn web search test, context editing cut token use by 84%. It also let agents finish tasks that would otherwise fail.

๐Ÿ’ก Good to know: These are Anthropic's own tests. Treat them as a guide, not a promise.

reported-results

Figure 5. Anthropic's own reported results, September 2025.

Memory frameworks: Mem0 and LangMem

In plain words: You don't have to build memory from scratch. Two popular options are Mem0 and LangChain's tools.

Mem0: a memory layer you plug in

  • You send Mem0 a conversation. It uses an AI model that "pulls out key facts, decisions, or preferences to remember."

  • Later, it finds the most relevant ones.

  • Each memory gets labels, such as a user_id, so every user's memories stay separate.

  • You can use Mem0's hosted service, or run its free, public code yourself.

LangChain: memory built into the framework

  • LangGraph is LangChain's framework for building agents. Memory is part of it.

  • Checkpointers save each chat, for example to your own Postgres or SQLite database.

  • Stores hold long-term memories that any chat can read.

  • LangMem adds tools to pull facts out of chats, keep user profiles, and let the agent improve its own instructions. That last one is procedural memory.

Other memory tools exist too, such as Letta and Zep. The same questions apply.

How to choose

  • What do you already use? If you build with LangGraph, its memory is already there.

  • Where must your data live? Both can run on your own servers.

  • What is the memory about? See Figure 6.

๐Ÿ’ก Good to know: Don't choose by benchmarks, the tests used to compare tools. On one, LOCOMO, Mem0's 2025 paper reports a 26% relative improvement over ChatGPT's memory, judged by an AI. That's Mem0 testing itself.

Choosing the right memory for your agent

In plain words: What your agent does decides what it needs to remember. Take a customer support agent.

  • Episodic memory: past tickets and chats.

  • Semantic memory: the customer's details, plan and preferences.

  • Knowledge: RAG over your help articles and refund policy.

  • Best fit: Mem0. The memory is mostly about the customer, and it must carry over between chats.

๐Ÿ’ก Rule of thumb: Pick Mem0 when the memory is about the user. Pick LangChain when it's about the work. Many systems use both.

Decision-chart

Figure 6. Which memory setup fits which agent. Editorial rule of thumb.

The memory lifecycle

In plain words: Good memory is picky about what it saves. It also forgets on purpose.

  • What to save: stable facts, decisions, corrections and outcomes.

  • What to skip: small talk, one-off details and guesses. "The customer seemed annoyed" is a guess, not a fact.

  • Note the source and date. Who said it, and when? Figure 7 shows an example.

  • When facts clash, the newer confirmed fact wins. Say the customer said "Basic plan" in March, but billing shows Pro since June. Pro wins. Mem0's newer version keeps old facts as history but ranks current ones higher.

  • Forget on purpose. Mem0 lets you give each memory an expiry date, but that only hides it. Anthropic's docs suggest you "periodically delete memory files that haven't been accessed in a long time." If a user asks you to delete their data, use Mem0's delete call.

saved-memory-labels

Figure 7. One saved memory and its labels.

๐Ÿ’ก Good to know: Some systems use several agents that work together. Give each one private memory plus shared memory, and decide who can write what. A billing agent might write payment facts that a chat agent can only read.

memory-lifecycle

Figure 8. The six steps of the memory lifecycle.

Privacy, security and common mistakes

In plain words: Memory holds personal data. Treat it with care.

  • Don't save sensitive data you don't need. Anthropic advises you to "add validation that strips sensitive data before your handler writes the file." In other words, check and clean the data before saving it.

  • Keep users and companies strictly apart. Filter every save and every lookup by user and company. Anthropic's Claude app keeps a separate memory for each project.

  • Let users see, fix and delete what's saved. Claude's app lets users view and edit a memory summary.

  • Watch for memory poisoning. Say an agent reads a web page with hidden instructions and saves them as a "fact". Now the attack lasts beyond that one chat.

๐Ÿ’ก Good to know: OWASP, a well-known security group, lists memory poisoning as ASI06 in its Top 10 for Agentic Applications. Only save facts from trusted sources, and record where each one came from.

Common mistake

What happens

Fix

Saving everything

Searches return clutter

Save facts, decisions and corrections only

Never forgetting

The store fills with old facts

Expiry dates and regular cleanup

Trusting old memories

The agent uses an old plan or address

Check the date, and confirm before acting

Mixing users' data

One customer sees another's details

Filter every lookup by user and company

Memory at a glance

  • Are your documents (knowledge) kept apart from user facts (memory)?

  • Do memories expire, and can users delete them?

  • Is every save and lookup limited to one user and one company?

  • Is text from web pages and emails checked before it becomes a memory?

Where to start

  1. Pick the one fact users repeat most, like their plan.

  2. Save it with a source, a date and a user ID.

  3. Load it at the start of each new chat.

  4. Add summarising once chats get long.

  5. Add expiry dates and a delete option.

Conclusion

An AI agent that forgets isn't broken. That's how language models work. Memory is the code you add around it. Get one fact right, then see if the complaints stop.

Frequently asked questions

Frequently asked questions

Q1. What is memory in AI agents?

Memory allows AI agents to store and retrieve information from past interactions, including user preferences, facts, and decisions.

Q2. What are the four types of memory in AI agents?

The four types are working, episodic, semantic, and procedural memory.

Q3. What is the difference between short-term and long-term memory in AI agents?

Short-term memory retains information within the current context window, while long-term memory stores information across conversations.

Q4. What are the best AI agent memory frameworks?

Mem0 and LangGraph with LangMem are popular frameworks for storing, retrieving, and managing AI agent memories.

Q5. How do you secure memory in AI agents?

Use data isolation, input validation, source tracking, expiry dates, and deletion controls to protect stored memories.

SHARE