When you first start working with Large Language Models (LLMs), you usually begin by writing a prompt. You ask a question, and the model provides an answer. This interaction feels magical, but quickly runs into limitations. What if the model needs access to company documents? What if it needs to know what was discussed ten minutes ago? Or what if it needs to be able to perform certain tasks, such as querying an API?
This is where the role of simple prompt engineering ends and the domain of context engineering begins. Context engineering is the art and science of dynamically building the working environment of an AI model. In this article, we explain why the complete context—instructions, documents, conversation history, and tools—determines what a model can do, and why more context, paradoxically, does not always result in better answers.
The distinction between the two terms is often not clearly drawn, but it is crucial in the development of AI applications.
Prompt engineering focuses on the exact formulation of the prompt. It is about choosing the right words, adding examples (few-shot prompting), and structuring the task so that the model understands exactly what is intended. You are working with the user's input.
Context engineering, on the other hand, is about the architecture of the information sent to the model along with the prompt. The model itself is 'static'; it knows nothing past the date its training stopped (the so-called knowledge cutoff). To make the model smart and relevant for a specific task, you must provide it with an up-to-date, structured, and filtered set of data. We call this entire set the context window.
Imagine asking a chef to bake a cake. Prompt engineering is saying: "Bake a fluffy chocolate cake in the shape of a star." Context engineering is ensuring the chef is in the kitchen, the ingredients are weighed and ready, a recipe is next to the oven, and the chef has a list of ingredients you are allergic to. Without that context, the chef simply cannot execute the perfect instruction.
When a developer sends an API request to a model (such as OpenAI's GPT-4, Google's Gemini, or Anthropic's Claude), the payload does not consist of a single sentence. It is a complex JSON object that has been carefully put together. We can roughly divide this context into four building blocks.
The foundation of your context is the system instruction. This is an instruction invisible to the end user, which defines the basic behavior, tone of voice, and safety restrictions of the AI. In the context hierarchy, this instruction carries a lot of weight. For example, a system instruction sets the boundaries: "You are a helpful customer service bot for an online store. Always reply in English. Never give medical advice. If you do not know the answer, refer the user to a human colleague."
By default, LLMs have no internal memory; they are stateless. Every request to an API is a completely new interaction. To create the illusion of a fluid conversation, the application must store the previous questions and answers and resend them with each new request. This requires active management, as we discuss in more detail in our article on memory in LLM apps. If the history becomes too long, you must summarize or omit old messages to stay within the limits.
Perhaps the most important part of modern context engineering is Retrieval-Augmented Generation, or the dynamic retrieval of information. Instead of cramming all company information into the model (which is impossible), you build a system that first analyzes the user's query, then searches for relevant documents in a database, and adds only those specific paragraphs to the context. Want to get started with this yourself? Then read our guide on RAG for beginners.
Nowadays, you can also include descriptions of external tools or APIs in the context (often in JSON schemas). For example, you tell the model: "Here is a function to retrieve the current weather forecast." This teaches the model that it does not have to guess the weather itself, but that it can return a signal to your application asking: "Execute this function for me and return the result."
Newer models offer gigantic context windows. Where the first generations could handle a maximum of 4,000 tokens (about 3,000 words), we now see models with context windows of 1 to even 2 million tokens (equivalent to thousands of pages of text). This sounds like the perfect solution: you just send your entire archive along and let the model figure it out. However, in practice, this often turns out to be a poor strategy. We call this the Context Paradox.
Just like humans, language models struggle to distribute their attention evenly across a massive amount of text. Researchers have identified a phenomenon called Lost in the Middle. Models are extremely good at remembering and using information that is at the very beginning or the very end of the context. However, information hidden halfway through a huge document is often overlooked or misinterpreted. To better understand how models prioritize information, knowledge of the underlying attention mechanism is essential.
If you add too much irrelevant information to the context, 'noise' is created. Suppose the user asks about the 2024 return policy, and you include documents from 2022, 2023, and 2024 in the context. The model must now actively resolve the contradictions. The more noise there is in the context, the greater the chance that the model will make mistakes and start hallucinating. You can find more insight into how and why models make things up in our overview on understanding hallucinations.
Processing tokens costs computing power, time, and money. The costs of LLM APIs are almost always calculated per processed token. If you unnecessarily send a hundred thousand tokens of background information with every query, your operational costs will skyrocket. In addition, the waiting time (latency) for the user will increase. The model simply needs more time to read a massive block of text before it can even start typing.
Successful AI applications distinguish themselves through efficient context engineering. They do not cram everything into the prompt, but use advanced techniques to keep the context as rich yet concise as possible.
| Technique | Goal | How it works |
|---|---|---|
| Reranking | Increase relevance | After retrieving documents (via vector search), a specialized 'reranker' model is used to resort the documents. Only the top 3 most relevant pieces actually enter the context. |
| Prompt Compression | Save tokens | Filtering unnecessary stop words, whitespace, or irrelevant sentences from raw data before presenting it to the model. |
| Context Caching | Cost & speed | Many API providers now offer 'Prompt Caching'. If you repeatedly send the same large document (for example, a huge legal code), the server temporarily stores its analysis. You then pay less, and the model responds much faster to follow-up questions about this document. |
To make context engineering tangible, it is useful to see how an application communicates with a language model under the hood. Instead of a simple string, the request is usually built from an array (a list) of messages, each with a specific role. Here, we look at a highly simplified JSON format, as seen in many frameworks (to build such architectures with Python, you can visit our external guide on LangChain basics).
[
{
"role": "system",
"content": "You are a helpful assistant. Answer questions solely based on the provided documents."
},
{
"role": "user",
"content": "Hello, can you tell me how to reset my password?"
},
{
"role": "assistant",
"content": "Sure! To reset your password, go to the settings..."
},
{
"role": "user",
"content": "And how long does it take before I receive the reset link?\n\n[CONTEXT-DOCUMENT-1]: The reset email is typically sent within 2 minutes."
}
]In the example above, you see the convergence of the three main roles:
The task of the context engineer is to determine how many of those old 'user' and 'assistant' messages are kept, how much weight the 'system' prompt carries, and how many '[CONTEXT-DOCUMENTS]' can be safely added without confusing the model.
Prompt engineering is like sending an email to an employee with a specific question. Context engineering is onboarding that employee: you give them a desk, a filing cabinet, a manual, and a set of company rules. The quality of your AI application stands or falls with this process.
Instead of relying on models with an ever-larger, unlimited context window, it pays to keep the context tight and controlled. A model that is fed exactly the right, clean, and relevant information delivers faster, cheaper, and more reliable answers than a model that has to search for a needle in a haystack of millions of tokens.
Want to get started designing smart systems around LLMs yourself? Then check out our external overview where you can compare different models and their context limits or dive deeper into our developer documentation.