In-context learning: how a model learns without weight updates

When a large language model (LLM) performs a task based on a few examples in the prompt, it appears to learn on the spot. Give the model three examples of a complex transformation — such as converting unstructured Dutch legal citations into a tight JSON schema — and by the fourth example it flawlessly follows the same pattern. Yet during this interaction, fundamentally nothing changes about the network's parameters. No backpropagation takes place, no gradients are calculated, and the static matrix values on disk or in VRAM remain completely identical.

This phenomenon is called in-context learning (ICL). Instead of physical changes to the neural network, the input text manipulates the transformer's intermediate activation states. In this article, we dissect the mathematical and mechanistic theories behind this process. We look at how induction heads copy patterns, how transformers simulate temporary optimization loops via implicit gradient descent, where the hard capacity limits lie, and how ICL relates to methods such as parameter-efficient fine-tuning.

The fundamental difference between training phases and inference

To understand why in-context learning is so remarkable, we first need to sharpen the distinction between parametric learning and activation-driven steering. During the classic training phase — that is, pre-training and supervised fine-tuning — hundreds of billions of tokens are passed through the network. A loss function measures the difference between the predicted token and the actual target token. Through backpropagation, gradients are computed back through all the layers, after which an optimizer such as AdamW updates the weights step by step.

To understand what these weights physically consist of, the article on parameters and weights in neural networks explains how numbers in matrices represent the final knowledge. Once a model is in production during inference, these weight matrices are in a strict read-only state (frozen weights).

In in-context learning, the static weights are not changed, but instead function as a fixed neural computer program. This program processes a series of input tokens and builds up a dynamic 'state' per layer in the so-called residual stream. The provided examples (few-shot demonstrations) serve as input data that modulates this internal state. The network computes a function $f(x)$ in which the function parameters are constant, but in which the context $C$ dynamically parameterizes the effective transformation from input to output.

Mechanistic explanation: the role of induction heads

One of the most concrete explanations for in-context learning comes from the field of mechanistic interpretability. Researchers have discovered that transformer models develop specific subcircuits that function as so-called induction heads. These are specialized attention mechanisms that consist of a collaboration between at least two consecutive attention layers.

Anyone who wants to dive deeper into how circuits in neural networks are scientifically isolated will find detailed methodologies in the overview on mechanistic interpretability in LLMs.

An induction head essentially performs an abstract search-and-copy operation over the context. The mechanism works in two steps:

• Layer 1 (Previous Token Head): This attention head looks at the current token $A$ and writes information about the immediately preceding token to the residual stream.
• Layer 2 (Induction Head): When the token $A$ reappears later in the context, this head looks up earlier instances of $A$, reads which token $B$ directly followed it, and increases the probability that token $B$ should be generated again after the current token $A$.

This induction algorithm $[A][B] \dots [A] \to [B]$ forms the microscopic basis for pattern recognition. When we provide a model with structured input-output examples (such as Input: Amsterdam -> Output: Noord-Holland), induction heads extremely quickly recognize the syntactic and semantic separators. They ensure that after seeing the prompt, the network immediately activates the corresponding mapping for a new element (such as Utrecht ->).

The mathematical theory: implicit gradient descent in the forward pass

Besides the mechanistic interpretation with induction heads, there is an influential theoretical hypothesis: transformers can, during the forward pass, simulate a form of implicit gradient descent . Various mathematical studies show that the linear transformations within multi-head attention layers can be structurally equivalent to one or more steps of optimization via linear regression.

In this model, the transformer architecture functions as a meta-optimizer. The model's weights are trained during pre-training in such a way that the activations in the early layers form an abstract representation of the dataset (the demonstrations in the context). The deeper layers then perform operations that mathematically amount to minimizing an internal objective function on those examples, even before the last token is generated.

Task Retrieval versus Task Learning

A crucial debate within AI research is whether a model, via ICL, actually learns a new task, or whether the examples merely serve to retrieve an already existing task from latent memory (task retrieval). If a model has already seen millions of pages of translations during pre-training, a prompt with three English-Dutch examples doesn't teach the network how to translate; it merely tells the network: now activate translation mode.

Researchers test this distinction with so-called 'flipped-label' experiments. Here, models are given examples in which the truth has been deliberately reversed (for example, a sentiment analysis where positive sentences get the label Negatief and vice versa). Small models often ignore this reversal and keep following their pre-trained associations (zero-shot prior). Very large models, on the other hand, override their internal preference and adapt to the reversed rules in the context. This proves that larger architectures are genuinely capable of algorithmic learning based on runtime instructions.

The influence of prompt engineering and demonstrations

The effectiveness of in-context learning depends heavily on how demonstrations are structured. Empirical analyses show that ICL is sensitive to four distinct factors in the input:

• The input format: The consistent structure (such as Vraag: ... Antwoord: ...) helps the attention heads clearly separate input and output fields.
• The distribution of the input: The model learns which terminology and concepts are relevant to the domain.
• The label space: By seeing examples, the network understands which categories are allowed (for example, only [Valide, Invalide, Onbekend]).
• The input-to-label mapping: The actual relationship between the specific question and the answer.

Remarkably, many mid-sized models already perform significantly better just by seeing the format and the allowed label space, even if some of the examples contain an incorrect label. For advanced logical deduction, however, correct input-output mapping is indispensable.

In the code snippet below, we demonstrate how a structured Python call for in-context learning can be set up with a deterministic evaluation on a Dutch entity recognition task:

import json

# Voorbeelden demonstreren het gewenste abstracte schema aan het model
few_shot_context = """Taak: Extraheer Nederlandse overheidsinstanties en afkortingen.

Invoer: Het Uitvoeringsinstituut Werknemersverzekeringen beoordeelt de aanvraag.
Uitvoer: {"instantie": "Uitvoeringsinstituut Werknemersverzekeringen", "afkorting": "UWV"}

Invoer: Volgens de Belastingdienst moet de aangifte voor mei binnen zijn.
Uitvoer: {"instantie": "Belastingdienst", "afkorting": null}

Invoer: De Sociale Verzekeringsbank keert de AOW maandelijks uit rond de 23e.
Uitvoer: {"instantie": "Sociale Verzekeringsbank", "afkorting": "SVB"}
"""

nieuwe_invoer = "Invoer: Het Centraal Bureau voor de Statistiek publiceert inflatiecijfers.\nUitvoer:"

volledige_prompt = f"{few_shot_context}\n{nieuwe_invoer}"
# Bij inferentie activeert deze structuur direct de juiste inductiepaden

Limits and vulnerabilities of in-context learning

Although in-context learning is exceptionally powerful, it has clear technical limitations that must be carefully weighed in production environments.

1. Order Sensitivity: The order in which examples are presented can cause a model's accuracy to fluctuate by tens of percentage points. If all the positive examples happen to be at the end of the prompt, the model shows a strong 'recency bias' and will disproportionately generate a positive answer.

2. Attention Dilution: As the context grows longer with dozens of examples, the attention weights have to be spread across thousands of tokens. This can cause the model to overlook subtle details in early examples, an effect known as the 'lost in the middle' syndrome.

Anyone designing practical systems who wants to understand how context lengths affect processing can consult the overview article on what a context window exactly entails and how limits work.

3. Computational Cost per Query: Unlike a fine-tuned model — which only needs the bare question after training — ICL requires that all demonstration tokens be resent and reprocessed with every individual API call. This increases memory usage in the KV cache and drives up latency and token costs.

When do you choose ICL versus Fine-Tuning or RAG?

In-context learning is not a universal replacement for other adaptation methods. Choosing the right architecture requires a clear trade-off between data volume, latency requirements, and the nature of the task.

Anyone unsure between different architectures for business applications can read the extensive trade-off analysis in the comparative article on the choice between fine-tuning, prompting, and RAG.

ICL is the superior choice when:

• Only a few (1 to 20) training examples are available;
• The task definition changes frequently and allows no downtime for retraining;
• You are using closed models via external APIs where weight updates are technically impossible.

Fine-tuning, on the other hand, is necessary when you have thousands of examples to teach a specific vocabulary or a complex writing style, or when the latency and cost of long prompts in production systems become unacceptably high.

Optimization in practice: Context Engineering

Because ICL depends entirely on the content of the context window, carefully structuring this space has grown into a discipline of its own. To discover how to structure prompts to get maximum steering out of induction circuits, the guide on context engineering and advanced prompt construction offers concrete best practices.

Selecting the most representative examples via semantic search algorithms (Dynamic Few-Shot Selection) ensures that only examples directly relevant to the user's specific question are loaded in. This makes optimal use of the available tokens without unnecessarily burdening the attention distribution.