# Mechanistic interpretability explained - llmnet.nl

[Skip to content](#lm-inhoud)Network/[NL](/en/mechanistisch-interpreteren-uitgelegd)EN[Hubhub.llmnet.nlCompare models on task, language, cost and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organization, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmechanistisch-interpreteren-uitgelegd&text=Mechanistic%20interpretability%20explained%20-%20llmnet.nl)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmechanistisch-interpreteren-uitgelegd)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmechanistisch-interpreteren-uitgelegd&title=Mechanistic%20interpretability%20explained%20-%20llmnet.nl)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmechanistisch-interpreteren-uitgelegd&text=Mechanistic%20interpretability%20explained%20-%20llmnet.nl)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmechanistisch-interpreteren-uitgelegd)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmechanistisch-interpreteren-uitgelegd&title=Mechanistic%20interpretability%20explained%20-%20llmnet.nl)[](#)

# Mechanistic interpretability explained

By Ivo Donker — compiled with AI support (Claude & Gemini) · Last updated: 6 August 2026

Large language models perform impressively on a wide range of tasks, from writing source code to answering complex knowledge questions. Yet the way these networks arrive at their answers largely remains hidden in a web of billions of numbers. Mechanistic interpretability is the research field that aims to change this. The central question of this field is not whether a model performs a task well, but exactly how the internal computation proceeds between the moment of input and the final output.

Instead of treating a model as a closed box whose results we merely measure, mechanistic interpretability tries to dissect the network the way a software engineer reads a compiled program. The goal is to expose the precise algorithms and internal structures that arose in the network during training.

## The difference from traditional explanation methods

Classic explainability methods in artificial intelligence focus mainly on the input. Well-known approaches analyze, for example, which words or image elements weighed most heavily in a specific decision. These techniques do show which parts of the text had influence, but they do not show how how the model processed that information.

Mechanistic interpretability approaches this differently. It does not look for a correlation between input signals and outcomes, but for the actual computational mechanism. It describes how individual components transform, pass on, and combine information. As discussed earlier in the article about [parameters and weights](https://leren.llmnet.nl/en/parameters-en-gewichten), the weights determine what a model knows and can do. Mechanistic research tries to decipher the exact logic embedded in those weights.

Core difference: Where traditional explainability asks "Which word did the model look at?", mechanistic interpretability asks "Which internal algorithm transformed word A into prediction B?"

## Why a single activation doesn't give a clear picture

An obvious idea is to simply look at when specific neurons in the network become active. One might hope that one neuron is responsible for the concept "France," and another for "capital." In practice, this rarely works. Neurons typically respond to a colorful mix of seemingly unrelated topics, such as both mathematical formulas and French poetry.

This is because the model doesn't compute in neatly delineated boxes, but in directions within a high-dimensional vector space. Concepts are represented as specific vectors or directions within that space. Because a single direction is formed by multiple neurons at once, the activation of a single neuron on its own is virtually unreadable to a human observer.

## Superposition: more concepts than directions

The phenomenon that causes this is known as superposition. During training, a network learns an enormous number of different concepts. However, the dimensionality of the internal representation — the number of available computational axes — is limited by the transformer's architecture. To work around this limitation, the model packs in more concepts than there are orthogonal (perpendicular) directions available.

In plain terms, superposition means the model shares directions. Two concepts that rarely occur together are stored at a very small angle relative to each other. Because they are nearly perpendicular, they barely interfere with each other. But to someone looking at the individual neurons from the outside, a single neuron therefore appears to respond to multiple unrelated things. The neuron, in fact, contributes to the construction of several shared directions at once.

Property | 
Memory per concept (Monosemantic) | 
Superposition (Polysemantic) | 

Representation | 
One neuron represents exactly one clear concept. | 
One neuron contributes to multiple concepts at once. | 

Capacity | 
At most as many concepts as there are neurons. | 
Far more concepts than the number of available neurons. | 

Interpretability | 
Directly readable and easy to understand. | 
Very complex; requires mathematical disentangling of directions. | 

## Disentangling entangled concepts

Because superposition complicates directly reading networks, researchers look for ways to undo this entanglement. The most commonly used route is deploying a separate helper model, a so-called sparse autoencoder. This helper model takes the entangled activations from the language model and projects them into a much larger, sparser representation.

In this expanded space, the concepts do become separated from each other. Each feature in the sparser representation now corresponds to a single, clear concept — for example, "a reference to the Eiffel Tower" or "a bracket that needs to be closed in C++ code." Through this transformation, the complex internally present information is finally translated into something humans can understand.

## The residual stream as shared working memory

To understand how information moves through a transformer, the concept of the residual stream is essential. You can think of the residual stream as the network's central highway or shared working memory. As described in the overview of [what a transformer is](https://leren.llmnet.nl/en/wat-is-een-transformer), a model consists of a series of consecutive layers.

Each layer in the transformer reads information from the residual stream, performs a computation, and then writes the result back onto the residual stream. The original information is preserved unless a later layer explicitly modifies or overwrites it. This explains why information about sentence structure or context is still present and retrievable thousands of tokens later.

Within this process, different components fulfill specific roles:

- Attention mechanisms: Move information between different positions in the text. As explained in the article about [attention](https://leren.llmnet.nl/en/attention-uitgelegd), these layers ensure that contextual relationships are established.

- Feed-forward networks: Process the gathered information per position and store factual associations and concepts.

- Residual stream: Acts as the continuous carrier to which all layers add their results.

## Circuits: the building blocks of model behavior

The central idea within the mechanistic approach is that a model performs tasks via circuits. A circuit is a small, cooperating network of specific components (such as certain attention heads and feed-forward layers) that together perform a recognizable subtask.

A well-known example of such a circuit is the "induction circuit." This is a combination of at least two attention heads that recognizes patterns of the type "if A was followed by B, and we now see A again, then B is likely to follow." Another example is a circuit that keeps track of people's names to correctly link pronouns. A complex language model consists of thousands of such small, overlapping circuits that operate in parallel and sequentially.

## Providing proof through intervention

Simply observing that a particular component is active during a specific task only demonstrates correlation. To truly prove that a mechanism is responsible for an outcome, researchers must intervene. This is done through causal interventions.

Researchers, for example, disable a specific component (so-called 'ablation') or replace the activation of a component with the activation from a different text input (so-called 'activation patching'). If the model's prediction then changes exactly as the hypothetical circuit predicts, that provides strong evidence that the mechanism has been correctly identified.

Causal interventions usually follow a fixed pattern:

- The model performs a task with a normal input, during which the internal activations are stored.

- The input is slightly modified, causing the outcome to change.

- Researchers gradually replace specific activations from the modified run with the original activations.

- As soon as the original outcome is restored, the exact component that carries the crucial information has been located.

## Limitations of interventions and the scaling problem

Although causal interventions provide stronger evidence than mere correlation, this approach is not infallible either. Neural networks are, after all, quite robust and often contain redundant paths. If a researcher disables one circuit, the model can sometimes still reach the correct answer via an alternative route (a backup system). This can mistakenly lead the researcher to conclude that the disabled component played no significant role.

In addition, the field faces an enormous scaling problem. So far, researchers have mainly succeeded in mapping small, well-defined subtasks in relatively small models. Dissecting a gigantic model with hundreds of billions of parameters requires enormous computing power and advanced automation. Manually analyzing every circuit is practically impossible with current large models.

## Practical applications: more than curiosity

Mechanistic interpretability is not a purely theoretical exercise; it has direct practical relevance for the safety and reliability of AI systems. Understanding how a model computes internally allows for steering in ways that aren't achievable through external training alone.

Important practical applications include:

- Targeted steering: Instead of retraining a model with expensive data, a specific unwanted mechanism can be directly adjusted or neutralized within the weights.

- Detecting concealed behavior: A model can be trained to give a desirable answer, while the internal representation shows that the answer is based on incorrect or misleading reasoning. In thorough testing such as [red teaming and safety testing](https://benchmark.llmnet.nl/en/red-teaming-en-veiligheidstests) this helps uncover hidden risks.

- Preventing hallucinations: By understanding the mechanism behind factual recall, it becomes clearer how the network separates genuine knowledge from guessing tokens. For a broader picture, also check out the page about [understanding hallucinations](https://leren.llmnet.nl/en/hallucinaties-begrijpen).

## The boundary between internal computation steps and generated explanation

A common misconception is that a language model can explain its own workings when you ask it "Why did you give this answer?" Techniques such as [chain-of-thought](https://leren.llmnet.nl/en/chain-of-thought) have the model write out its reasoning step by step. However, this is a textual representation generated after the fact or incidentally, not a faithful account of the actual computational steps in the network.

A language model that describes its own reasoning produces a plausible-sounding story that fits the context. This story can be useful, but it does not guarantee that the weights and circuits actually followed that exact route. Mechanistic interpretability therefore does not look at what the model says about its behavior, but at what the numbers in the network actually do.

## What this means for users and developers

For users and organizations implementing AI applications, mechanistic research makes clear that explainability is far from a solved problem. Claims by vendors about 'fully transparent' or 'understandable' AI systems should be interpreted with caution.

Science is steadily making progress in understanding small building blocks, but scaling these techniques up to full, complex models is a long-haul process. As long as we cannot fully grasp the internal computational mechanisms, monitoring and testing actual behavior remains an indispensable part of responsible AI use.

## Further reading

- [Attention explained](https://leren.llmnet.nl/en/attention-uitgelegd)

- [Parameters and weights](https://leren.llmnet.nl/en/parameters-en-gewichten)

- [What is a transformer](https://leren.llmnet.nl/en/wat-is-een-transformer)

- [Embeddings explained](https://leren.llmnet.nl/en/embeddings-uitgelegd)

- [Understanding hallucinations](https://leren.llmnet.nl/en/hallucinaties-begrijpen)

- [Chain-of-thought](https://leren.llmnet.nl/en/chain-of-thought)

- [Red-teaming and safety testing at benchmark.llmnet.nl](https://benchmark.llmnet.nl/en/red-teaming-en-veiligheidstests)

llmnet.nl - learning and explanations about language models