# Memory in LLM apps: what does the model remember?

[Skip to content](#lm-inhoud)Network/[NL](/en/geheugen-in-llm-apps-wat-de-model-onthoudt-en-wat-jij-moet-doen)EN[Hubhub.llmnet.nlCompare models on task, language, cost and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organization, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fgeheugen-in-llm-apps-wat-de-model-onthoudt-en-wat-jij-moet-doen&text=Memory%20in%20LLM%20apps%3A%20what%20does%20the%20model%20remember%3F)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fgeheugen-in-llm-apps-wat-de-model-onthoudt-en-wat-jij-moet-doen)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fgeheugen-in-llm-apps-wat-de-model-onthoudt-en-wat-jij-moet-doen&title=Memory%20in%20LLM%20apps%3A%20what%20does%20the%20model%20remember%3F)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fgeheugen-in-llm-apps-wat-de-model-onthoudt-en-wat-jij-moet-doen&text=Memory%20in%20LLM%20apps%3A%20what%20does%20the%20model%20remember%3F)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fgeheugen-in-llm-apps-wat-de-model-onthoudt-en-wat-jij-moet-doen)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fgeheugen-in-llm-apps-wat-de-model-onthoudt-en-wat-jij-moet-doen&title=Memory%20in%20LLM%20apps%3A%20what%20does%20the%20model%20remember%3F)[](#)

 
# Memory in LLM apps: what the model remembers and what you need to do

 By Ivo Donker — compiled with AI assistance (Claude & Gemini)

 
 
### What you need to know beforehand

 This is module 3, working with your own data. To fully understand the architecture of external memory, it helps to understand how the underlying neural networks handle input. Refer to the article on [what a transformer exactly does](https://leren.llmnet.nl/en/wat-is-een-transformer) for the basic mechanics of attention vectors. If you're unsure about technical terms such as embeddings, context windows, or parameters, first check out the complete [AI glossary with definitions](https://leren.llmnet.nl/en/ai-begrippenlijst) to be able to follow the theory smoothly.

 

 A common misconception among beginning software developers is that a large language model (LLM) learns from the conversations held with it. When a user asks: "Do you remember what I just said?", the model appears to recall this flawlessly. In reality, an LLM has no memory of its own between two API calls. Every language model is inherently a pure, mathematical function that converts input tokens into output tokens: completely stateless.

 The illusion of an ongoing conversation arises solely because the surrounding application repackages and resends the entire conversation history with every new interaction. As applications become more complex — from simple customer service bots to autonomous software agents — simply sending along all previous text falls short. Context windows fill up, latency rises exponentially, and API costs explode. In this article, we cover why language models are stateless and how developers build external memory systems to guarantee continuity, personalization, and reliability.

 
## The fundamental illusion: stateless inference and parameters

 To understand what an application needs to solve, let's first look at what the model does and doesn't retain. A trained language model consists of hundreds of billions of static weights. These weights contain 'parametric memory': condensed representations of patterns, facts, grammar, and reasoning structures from the training dataset. Once the model runs in production, these weights are frozen.

 When a request (inference request) is sent to the model API, the following happens:

 1. The input text is converted into numerical tokens.
 2. The transformer performs matrix multiplications on these tokens via the attention mechanism.
 3. The model calculates probability distributions for the next tokens and generates a response.
 4. As soon as the response is complete, all intermediate activation states are cleared from video memory (VRAM).

 The model doesn't save anything to disk and doesn't adjust its own weights. Anyone who sends a new message a minute later without context starts with a completely clean slate as far as the model is concerned. As an application builder, you are therefore responsible for designing a 'non-parametric memory': an external data layer that stores and structures relevant information, and selectively injects it into the context window at the right moment. For anyone wanting to make the move to advanced autonomous systems, this is a core skill; read more about it in the guide on how to become a successful [AI agent engineer](https://leren.llmnet.nl/en/ai-agent-engineer-worden-2026) and build robust software loops.

 
## Short-term context management: sliding windows and token budgets

 The most basic form of application memory is storing the chat history in a traditional database (such as PostgreSQL or Redis) and injecting this list of messages into the API's messages field on every turn. We call this short-term session memory. However, this immediately runs into the limit of the context window.

 To prevent the token limit from being exceeded, developers use sliding-window strategies. Here, only the most recent $k$ interactions are kept. While this is simple to implement, it has major drawbacks: information mentioned $k+1$ turns ago abruptly disappears from the model's view. For this reason, modern systems work with dynamic token budgets.

 
 
 
 
 Strategy | 
 Advantages | 
 Disadvantages | 
 Ideal use case | 
 

 
 
 
 Fixed message limit (Buffer Window) | 
 Minimal compute time, trivial to build | 
 Doesn't account for token length; sudden context loss | 
 Short, transactional chatbots | 
 

 
 Dynamic Token Budget (FIFO) | 
 Optimal filling of the context window | 
 Older context still drops away without compression | 
 Helpdesk assistants with medium-length sessions | 
 

 
 Summarizing Buffer (Summary Memory) | 
 Retains the main outline over very long sessions | 
 Extra LLM call needed; loss of detail | 
 Complex consulting engagements, co-pilots | 
 

 
 Hybrid Semantic Memory | 
 Combines recent turns with targeted vector retrieval | 
 Complex orchestration and higher latency | 
 Autonomous agents, long-term personal assistants | 
 

 
 
 

 In a dynamic token budget, we assign hard limits to different components in the prompt. A typical budget breakdown looks like this:

 # Voorbeeld van een tokenbudget-configuratie in Python
TOTAL_CONTEXT_LIMIT = 8192
SYSTEM_PROMPT_RESERVE = 1000
OUTPUT_GENERATION_RESERVE = 1500
DYNAMIC_MEMORY_BUDGET = TOTAL_CONTEXT_LIMIT - SYSTEM_PROMPT_RESERVE - OUTPUT_GENERATION_RESERVE
# Beschikbaar voor sessiegeschiedenis en opgevraagde feiten: 5692 tokens

 When the conversation history exceeds the budget of 5692 tokens, the application has to make choices: which old messages are deleted, compressed, or moved to a long-term archive? This directly touches on advanced context design; see the in-depth article on [how context engineering works in production environments](https://leren.llmnet.nl/en/context-engineering-uitgelegd) for concrete patterns to make optimal use of tokens.

 
## Summarization strategies: incremental compression

 When a conversation lasts longer than a few dozen interactions, discarding old messages via FIFO (First In, First Out) is often unacceptable. If a user mentions in turn 2 that they're allergic to nuts, a recipe assistant must not have forgotten that by turn 40. The solution to this is incremental summarization (rolling summarization).

 With incremental compression, a separate LLM task runs in the background as soon as the number of tokens passes a threshold value. This task takes the existing summary and the oldest unprocessed messages, and distills a new, updated summary from them. The current context window then contains:

 [Systeemprompt met kerninstructies]
[Lopende samenvatting van het eerdere gesprek (bijv. 250 tokens)]
[Recente interacties woordelijk (bijv. de laatste 5 berichten)]
[Nieuwste gebruikersvraag]

 This mechanism, however, brings specific trade-offs with it. First, generating the summary costs extra tokens and time. Second, 'information erosion' occurs: details that don't seem directly relevant to the summarizing LLM are left out and are thereby permanently lost to the active memory. That's why it's crucial to keep a sharp distinction between session context and persistent memory. In the article on [the conceptual difference between memory and context](https://leren.llmnet.nl/en/geheugen-en-context-verschil-uitgelegd) goes deeper into why compression is not the same as remembering.

 
## Long-term memory: vector databases and semantic retrieval

 For applications that need to function across days, weeks, or months, a linear summary isn't enough. Here, the software architecture introduces a vector database (such as Qdrant, Chroma, or pgvector) as external long-term memory. This process closely resembles Retrieval-Augmented Generation (RAG), but with one important difference: instead of static documents, the system indexes the interactions, facts, and preferences of the user themselves.

 The storage and retrieval process takes place in three phases:

 1. Extraction and Chunking: After every interaction, an extraction prompt analyzes whether any durable facts have been mentioned (for example: "User prefers TypeScript over Python"). These facts are cut loose from the conversational noise.
 2. Embedding: The fact is converted into a vector (a list of numbers) by an embedding model and stored in the vector database, tagged with metadata such as a timestamp and user ID.
 3. Semantic retrieval (Retrieval): For a new question, the application searches the database for vectors that are semantically close to the current user question via cosine similarity. The top-k most relevant memories are injected into the prompt.

 // Voorbeeld van een JSON-geheugenrecord in een vector database
{
 "id": "mem_982341",
 "user_id": "usr_4402",
 "timestamp": "2026-08-15T10:14:22Z",
 "fact": "Klant wil uitsluitend communiceren in het Nederlands en heeft voorkeur voor beknopte antwoorden.",
 "category": "preference",
 "importance_score": 0.85,
 "embedding": [0.0124, -0.0841, 0.0512, ...]
}

 The big advantage of semantic retrieval is scalability: a vector store can hold millions of interactions without filling up the LLM's context window. The downside is that semantic similarity doesn't always equal relevance. If a user asks: "What did we eat yesterday?", a naive vector memory searches for messages about food, but may miss the temporal context if the date isn't explicitly included in the search query.

 
## Structured entity memory and Knowledge Graphs

 Vectors are excellent at finding vague concepts, but weak at exact relationships. If a user states: "My sister Marieke has two cats, Tom and Jerry", and three weeks later asks: "How many animals does my sister have?", a vector search can fail if the semantic distance between the chunks is too large.

 Advanced architectures therefore combine vector storage with structured entity memories or Knowledge Graphs. Here, a model explicitly parses entities and their mutual relationships into triples:

 (Gebruiker) — [HEEFT_ZUS] —> (Marieke)
 (Marieke) — [BEZIT] —> (Tom: Kat)
 (Marieke) — [BEZIT] —> (Jerry: Kat)

 By storing relational data in a graph database (such as Neo4j) or a structured relational table, the application can run deterministic queries before the context is handed to the language model. This prevents hallucinations about hard facts and ensures consistent personalization over long timelines.

 
## Memory architectures in practice: MemGPT and agentic loopback

 In 2023 and 2024, groundbreaking approaches emerged such as MemGPT (Memory-GPT), which mimic the workings of a traditional computer operating system for LLMs. In this paradigm, the model's context window is treated as working memory (RAM), while external databases function as the hard disk (Disk Storage).

 In such a system, the model is given special tools (functions) with which it can actively manage its own memory. Instead of the middleware passively trying to guess what's important, the model decides for itself:

 • core_memory_append(key, value): add a core fact to the always-present working memory.
 • archival_memory_insert(content): store a detailed memory in the external vector storage.
 • archival_memory_search(query, page): search the archive in a targeted way when more background knowledge is needed.

 This shifts control to the model itself. The model recognizes when an instruction is crucial for the future and autonomously calls a function to record it. The challenge here is reliability: small models sometimes forget to call functions, or pollute their memory with irrelevant details if the guiding system prompt hasn't been carefully optimized.

 
## Privacy, data retention, and forgetting: TTL and GDPR compliance

 An aspect that is often overlooked when designing memory in AI apps is data management and privacy. As soon as an application extracts and stores personal data in external vector or graph databases, this falls under legislation such as the General Data Protection Regulation (GDPR).

 A robust memory system must therefore have the following mechanisms:

 • Right to erasure (Right to be Forgotten): Users must have the ability to view, correct, and fully erase their memory profile. In a vector database, this requires that every embedding record be linked to a unique user_id and can easily be deleted via metadata.
 • Time-to-Live (TTL) and memory decay: Just like the human brain, an AI application must let less relevant memories fade over time. By applying a 'decay function' based on the age and usage frequency of a memory, you prevent outdated facts (such as an old temporary address) from polluting current answers.
 • Separation of sensitive data: Medical data, passwords, or payment information must be filtered out before they end up in a permanent memory archive.

 
## Evaluation and quality assurance of memory mechanisms

 How do you know whether a memory system actually performs as intended? Evaluating memory in LLM applications requires a systematic approach. A common mistake is manually testing a handful of prompts, which gives a false sense of certainty.

 For reliable operation, three core metrics are measured:

 1. Retrieval Precision & Recall: For a specific question, does the system retrieve exactly the historical facts needed to answer the question, without unnecessary clutter?
 2. Memory Retention Rate: Does a fact remain correctly stored after 10, 50, and 100 consecutive conversation turns, or does degradation occur in the summary?
 3. Latency & Token Overhead: What is the extra delay and cost increase that the memory layer adds to every user turn?

 To determine which memory prompt or search setting performs best, experimentation is necessary. Read in the overview on [systematically A/B testing prompts](https://benchmark.llmnet.nl/en/ab-testen-prompts) how to set up statistically valid comparisons to prevent regressions in your application.

 
## Conclusion and implementation trade-offs

 A language model remembers nothing on its own. It is a purely mathematical calculator that starts from zero with every call. The magic of long-term interactions, consistent assistants, and learning agents arises entirely in the architecture that you, as a developer, build around the model.

 When building an application, always start with the simplest solution: a dynamically calculated token budget with a buffer of recent messages. Only add an automated summarization layer or an external vector database once the use case explicitly calls for continuity across multiple sessions. Always ensure strict metadata filtering, respect privacy guidelines, and continuously measure whether the retrieved memories actually contribute to a better answer.

 
 
### Next up

 Do you want to make the move from static memory buffers to complex systems in which models independently call tools and make decisions? Then continue reading the article on [how AI learns to remember and forget in applications](https://leren.llmnet.nl/en/geheugen-in-llm-apps) for additional software patterns and architectural blueprints.