# Self-attention dissected mathematically: dot product to softmax

[Skip to content](#lm-inhoud)Network/[NL](/en/self-attention-wiskundig-ontleed-van-dot-product-naar-softmax)EN[Hubhub.llmnet.nlCompare models on task, language, cost and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organization, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fself-attention-wiskundig-ontleed-van-dot-product-naar-softmax&text=Self-attention%20dissected%20mathematically%3A%20dot%20product%20to%20softmax)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fself-attention-wiskundig-ontleed-van-dot-product-naar-softmax)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fself-attention-wiskundig-ontleed-van-dot-product-naar-softmax&title=Self-attention%20dissected%20mathematically%3A%20dot%20product%20to%20softmax)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fself-attention-wiskundig-ontleed-van-dot-product-naar-softmax&text=Self-attention%20dissected%20mathematically%3A%20dot%20product%20to%20softmax)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fself-attention-wiskundig-ontleed-van-dot-product-naar-softmax)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fself-attention-wiskundig-ontleed-van-dot-product-naar-softmax&title=Self-attention%20dissected%20mathematically%3A%20dot%20product%20to%20softmax)[](#)

 
# Self-attention mathematically dissected: from dot product to softmax

 By Ivo Donker — compiled with AI assistance (Claude & Gemini) · 22 August 2026

 This is module 5, under the hood. Where conceptual articles often describe self-attention as a search engine in which tokens look at each other, in this article we dissect the exact linear algebra and matrix operations behind the mechanism. We follow an input vector from the initial projections to the weighted summation after the softmax layer.

 
 
### What you need to know first

 
 
- For an intuitive introduction, first read [how the attention mechanism works conceptually](https://leren.llmnet.nl/en/attention-uitgelegd).
 
- Understand how order information is linked to vectors via [how positional encodings work](https://leren.llmnet.nl/en/positional-encoding-uitgelegd).
 
- Basic knowledge of matrix multiplication, transposition, and vector spaces is required to follow the derivations.
 
 

 
## 1. The fundamental mathematical equation

 The core of the modern transformer architecture rests on one compact matrix formula introduced in 2017: Scaled Dot-Product Attention. In pure mathematical notation, the operation looks as follows:

 Attention(Q, K, V) = softmax( (Q * K^T) / sqrt(d_k) ) * V

 In this equation, Q (Queries), K (Keys), and V (Values) represent two-dimensional matrices derived from the input tokens. The scalar d_k represents the dimension of the key vectors. To understand why this specific sequence of operations produces context-aware representations, we need to break down each term into its individual dimensions and geometric meaning.

 Let's assume we have an input sequence of N tokens. Each token is represented by a dense vector with dimension d_model (for example, 4,096 in Llama 3 8B). The complete input matrix X therefore has dimension (N, d_model). Before any interaction between tokens takes place, the network transforms this input via three separate weight matrices.

 
## 2. Projection into Query, Key, and Value spaces

 A transformer does not calculate the relationships directly on the raw input vectors X. Instead, the model learns three linear transformations to construct separate representations for three specific roles:

 
 
- Query (Q): What is the current token looking for in the rest of the sentence?
 
- Key (K): What properties does this token offer to other tokens that are looking for information?
 
- Value (V): What actual substantive information does this token convey if there is a match?
 

 The transformations are carried out through matrix multiplications with learned weight matrices W_Q, W_K and W_V:

 Q = X * W_Q waarbij W_Q de vorm (d_model, d_k) heeft
K = X * W_K waarbij W_K de vorm (d_model, d_k) heeft
V = X * W_V waarbij W_V de vorm (d_model, d_v) heeft

 In standard multi-head architectures, it is nearly always true that d_k = d_v = d_model / h, where h represents the number of attention heads. If d_model = 4096 and h = 32, then d_k = 128. The resulting matrices Q, K and V each therefore have dimension (N, d_k).

 
 
 
 
 Matrix | 
 Mathematical dimension | 
 Example (N=4, d_model=4096, d_k=128) | 
 Function in the network | 
 

 
 
 
 X | 
 (N, d_model) | 
 (4, 4096) | 
 Embedded tokens including position information | 
 

 
 W_Q, W_K | 
 (d_model, d_k) | 
 (4096, 128) | 
 Projection weights for queries and keys | 
 

 
 W_V | 
 (d_model, d_v) | 
 (4096, 128) | 
 Projection weights for the content values | 
 

 
 Q, K, V | 
 (N, d_k) | 
 (4, 128) | 
 Projected representations per token | 
 

 
 
 

 
## 3. The inner product (dot product) as an affinity measure

 The next step is determining the pairwise affinity between each token. This is done by multiplying the query matrix Q by the transposed key matrix K^T:

 S = Q * K^T

 Because Q has shape (N, d_k) and K^T has shape (d_k, N), the result is a square matrix S with dimension (N, N). Each element S_{i,j} in this matrix represents the inner product (dot product) between the query vector of token i and the key vector of token j:

 S_{i,j} = q_i · k_j = som(q_{i,m} * k_{j,m}) voor m = 1 tot d_k

 Geometrically, this inner product measures the directional alignment between the two vectors in the d_k-dimensional space. If two vectors point in the same direction, this results in a high positive score. If they are perpendicular to each other, the score is exactly zero. If they point in opposite directions, a negative score results.

 Let's look at a concrete Dutch example with three tokens: ["De", "bank", "kraakte"]. If the token "kraakte" acts as a query, its vector is looking q_3 for a subject that can physically break. The key vector k_2 of the polysemous word "bank" (seating furniture vs. financial institution) has, in the context of furniture, a high similarity to q_3. The dot product q_3 · k_2 therefore produces a significantly higher value than q_3 · k_1 ("De").

 
## 4. Why scale by sqrt(d_k)? The variance derivation

 A crucial part of the formula is the division by sqrt(d_k). Without this scaling factor, this is referred to as Dot-Product Attention; with this factor, it is called Scaled Dot-Product Attention. Why is this specific square root necessary?

 Let's consider two random vectors q and k with length d_k, where we assume the individual components are independent random variables with a mean (expected value) of 0 and a variance of 1:

 E[q_m] = 0, Var(q_m) = 1
E[k_m] = 0, Var(k_m) = 1

 The inner product is the sum of d_k such products: Z = som_{m=1}^{d_k} (q_m * k_m). The expected value of each product q_m * k_m is 0. Because the terms are independent, we are allowed to sum the variances:

 Var(q_m * k_m) = Var(q_m) * Var(k_m) = 1 * 1 = 1
Var(Z) = Var( som_{m=1}^{d_k} (q_m * k_m) ) = som_{m=1}^{d_k} Var(q_m * k_m) = d_k

 The standard deviation of the dot product Z therefore grows proportionally to sqrt(d_k). With a modern dimension of d_k = 128 the unscaled dot product has a variance of 128 and a standard deviation of about 11.31. This means raw scores regularly take on values above +25 or below -25.

 When we send such extreme numbers through a softmax function, exponential saturation occurs. The largest value gets a probability of nearly 1.0, while all other positions converge to 0.0. In this saturated region, the gradient (the derivative) of the softmax function is nearly zero (the vanishing gradient problem). As a result, the training process via backpropagation stagnates completely. By dividing by sqrt(d_k) we normalize the variance of the softmax's input back to exactly 1:

 Var( Z / sqrt(d_k) ) = (1 / d_k) * Var(Z) = (1 / d_k) * d_k = 1

 
## 5. Softmax normalization and numerical stability

 The scaled affinity scores are normalized per row using the softmax function. For each row i in the affinity matrix S' = (Q * K^T) / sqrt(d_k) we calculate the attention weights A_{i,j}:

 A_{i,j} = exp(S'_{i,j}) / som_{l=1}^N exp(S'_{i,l})

 The matrix A again has dimension (N, N). Softmax guarantees two essential mathematical properties:

 
 
- Each weight A_{i,j} lies strictly between 0 and 1: 0 <= A_{i,j} <= 1.
 
- The sum of all weights in a row is exactly 1: som_{j=1}^N A_{i,j} = 1.
 

 In practical software implementations (such as PyTorch, CUDA kernels, or C++), a direct calculation of exp(x) can lead to numerical overflow if x is large. That's why the safe softmax trick is universally used, where the maximum value of the row is first subtracted from all elements before the exponent is calculated:

 m_i = max_j (S'_{i,j})
A_{i,j} = exp(S'_{i,j} - m_i) / som_{l=1}^N exp(S'_{i,l} - m_i)

 Because exp(a - c) / exp(b - c) = exp(a) / exp(b) this subtraction does not change the mathematical outcome, but it prevents intermediate results in floating-point representation (such as FP16 or BF16) from turning into +Infinity or NaN.

 
## 6. Weighted summation of the Value vectors

 After the weight matrix A with dimension (N, N) has been calculated, the final multiplication follows with the Value matrix V with dimension (N, d_v):

 Output = A * V

 The result is a matrix of shape (N, d_v). For each individual token i the final representation is a weighted linear combination of all value vectors in the sequence:

 Output_i = som_{j=1}^N (A_{i,j} * v_j)

 Here we see the mechanical synthesis of context: if token 1 ("kraakte") assigns a high attention score A_{1,2} = 0.85 to token 2 ("bank"), then 85% of the resulting vector for token 1 consists of the properties stored in the value vector v_2. The token has enriched its own representation with information from its contextual neighbors.

 
## 7. Causal masking in autoregressive decoders

 In decoder-only architectures such as GPT-4, Llama, or Claude, a token may not look ahead at future tokens during generation. A model predicting the fourth word may only pay attention to positions 1, 2, and 3.

 Mathematically, this is solved by adding a masking matrix before the softmax step M to the affinity matrix S':

 S'_{masked} = S' + M

 Where the mask M is defined as follows:

 M_{i,j} = 0 voor j <= i (huidige en eerdere tokens)
M_{i,j} = -inf voor j > i (toekomstige tokens)

 Because exp(-inf) = 0, the softmax operation results, for all future positions, j > i in an attention weight of exactly 0. The gradients likewise do not flow back through masked positions during backpropagation. In systems that perform inference, this mechanism allows us to cache earlier keys and values; see [how the structure of KV caching in transformer architectures](https://leren.llmnet.nl/en/kv-caching-opbouw) prevents earlier tokens from having to be recalculated every time.

 
## 8. A complete step-by-step numerical example

 To see all the steps together, we'll work through a minimalist example with N = 2 tokens and a dimension d_k = 2.

 Suppose the projected matrices Q, K and V contain the following values after transformation:

 Q = [ [1.0, 0.0], K = [ [1.0, 1.0], V = [ [2.0, 0.0],
 [0.0, 2.0] ] [0.0, 1.0] ] [1.0, 3.0] ]

 Step 1: Matrix multiplication Q * K^T

 K^T = [ [1.0, 0.0],
 [1.0, 1.0] ]

Q * K^T = [ [1.0*1.0 + 0.0*1.0, 1.0*0.0 + 0.0*1.0],
 [0.0*1.0 + 2.0*1.0, 0.0*0.0 + 2.0*1.0] ]
 = [ [1.0, 0.0],
 [2.0, 2.0] ]

 Step 2: Scaling by sqrt(d_k) = sqrt(2) ≈ 1.4142

 S' = (Q * K^T) / 1.4142
 = [ [0.7071, 0.0000],
 [1.4142, 1.4142] ]

 Step 3: Softmax per row

 For row 1: exp(0.7071) ≈ 2.0281, exp(0.0) = 1.0000. Sum = 3.0281.

 A_{1,1} = 2.0281 / 3.0281 ≈ 0.670
A_{1,2} = 1.0000 / 3.0281 ≈ 0.330

 For row 2: exp(1.4142) ≈ 4.1132, exp(1.4142) ≈ 4.1132. Sum = 8.2264.

 A_{2,1} = 4.1132 / 8.2264 = 0.500
A_{2,2} = 4.1132 / 8.2264 = 0.500

A = [ [0.670, 0.330],
 [0.500, 0.500] ]

 Step 4: Multiplication with V

 Output = A * V
Output_1 = 0.670 * [2.0, 0.0] + 0.330 * [1.0, 3.0] = [1.340 + 0.330, 0.000 + 0.990] = [1.670, 0.990]
Output_2 = 0.500 * [2.0, 0.0] + 0.500 * [1.0, 3.0] = [1.000 + 0.500, 0.000 + 1.500] = [1.500, 1.500]

Output = [ [1.670, 0.990],
 [1.500, 1.500] ]

 
## 9. Multi-Head Attention: parallelization and projection

 Instead of performing one large attention calculation with dimension d_model, Multi-Head Attention (MHA) splits the representation across h independent heads. Each head calculates its own attention distribution in a lower dimension d_k = d_model / h:

 head_i = Attention(Q * W_Q^i, K * W_K^i, V * W_V^i)
MultiHead(Q, K, V) = Concat(head_1, head_2, ..., head_h) * W_O

 The concatenation pastes the h matrices of shape (N, d_v) side by side into a single matrix of shape (N, h * d_v) = (N, d_model). The output projection matrix W_O (with dimension (d_model, d_model)) then linearly mixes all the heads.

 Why is this mathematically superior to a single head? With one head, a token can only direct its attention according to one distribution. With 32 or 64 heads, head 1 can focus on grammatical dependencies (verb-subject relations), head 2 on anaphora ("he" refers to "the minister"), and head 3 on semantic clustering.

 
## 10. Computational complexity and memory bottlenecks

 The computational and memory complexity of standard self-attention forms the biggest infrastructural bottleneck for large language models. Let's analyze the complexity in terms of FLOPs and memory usage:

 
 
- Matrix multiplication Q * K^T: Required 2 * N^2 * d_k operations.
 
- Softmax and storage of affinities: Has a time and memory complexity of O(N^2).
 
- Matrix multiplication A * V: Required 2 * N^2 * d_v operations.
 

 For a context length of N = 2.048 tokens, the A matrix contains about 4.19 million elements per head per layer. With a context of N = 128.000 tokens, this explodes to 16.38 billion elements per layer. Without optimizations, this leads to out-of-memory errors on GPUs (OOM).

 To break through this quadratic bottleneck in practice, modern hardware uses specialized algorithms; see how [FlashAttention speeds up calculations through smart GPU memory management](https://leren.llmnet.nl/en/flashattention-ontleed-snellere-berekening-via-gpu-geheugen) through online softmax and tiling. In addition, techniques at the inference level are used to reduce overall latency; take a look, for example, at the developments around [accelerating inference through speculative decoding](https://nieuws.llmnet.nl/en/speculative-decoding-versnelling-van-llm-inferentie).

 
 
### Continue reading with

 
 
- Deepen your knowledge of memory optimization at the architecture level via [grouped-query attention and memory savings](https://leren.llmnet.nl/en/grouped-query-attention-uitgelegd).
 
- Discover how hardware implementations tackle the computational bottleneck in [the in-depth analysis of FlashAttention](https://leren.llmnet.nl/en/flashattention-ontleed-snellere-berekening-via-gpu-geheugen).