Positional Encoding Explained: How Transformers Understand Order

Modern large language models are based on the transformer architecture. These models process enormous amounts of text and generate responses that come across as remarkably human. A fundamental part of how a transformer works is the way the model handles word order. Without a specific mechanism called positional encoding a transformer would be completely blind to the structure and order of a sentence.

In this article, we explain why the basic component of a transformer has no built-in sense of order, what goes wrong when position information is missing, which mathematical solutions have been developed for this, and what this means in practice for using language models.

The problem: Attention is permutation-invariant

To understand why positional encoding is necessary, we need to look at the core of what drives a transformer : the self-attention mechanism. When a text is entered, the words are first converted into tokens via tokenization. These tokens are then translated into vectors, or embeddings.

The attention mechanism calculates the relationship between every pair of tokens in a text. It compares the vectors of token A and token B using an inner product (dot product). The mathematical operation taking place here treats the input as a collection of separate elements rather than a sequential series. In mathematics, this property is called permutation invariance.

This means that if you feed the tokens into the attention layer in any random order, the calculated relationships between the individual tokens remain exactly the same. The model knows which words are present, but has no idea whatsoever which word comes first, second, or last.

Older models, such as Recurrent Neural Networks (RNNs), processed text step by step from left to right. As a result, order was automatically built in. Transformers, however, process all tokens simultaneously to enable parallel computation on graphics cards. That parallelism delivers enormous speed gains, but erases the natural sense of order.

A concrete example: The importance of word order

Language is highly dependent on order. The same set of words can take on a completely different meaning when the order changes. Suppose we feed the following two sentences into a model without positional encoding:

Both sentences contain exactly the same tokens: "The", "dog", "bites", "the", "man". For a standard attention operation without position information, sentences A and B are mathematically identical. The vectors for "dog" and "man" are assigned exactly the same attention scores relative to "bites" in both cases.

The result is that the model cannot distinguish the crucial difference between the actor and the object. Without a signal indicating which token is at position 2 and which token is at position 5, the model cannot correctly interpret the semantic meaning of the sentence. Positional encoding adds this essential signal to the token embeddings before they reach the first attention layer.

The evolution of Positional Encoding techniques

Over the years, various methods have been developed to add position information to transformers. We discuss the main approaches, the reasoning behind them, and their practical properties.

1. Fixed Sinusoidal Encodings (Vaswani et al., 2017)

The original transformer paper chose a fixed mathematical formula based on sine and cosine functions at different frequencies. A unique position vector is generated for each position in the text. This vector is simply added to the token's existing embedding.

• The idea: By using waves of different frequencies, each position gets a unique pattern. Nearby positions have vectors that closely resemble each other, while positions far apart differ clearly.
• Why it was devised: No extra parameters and weights needed to be trained for position. In addition, the researchers hoped this would allow the model to extrapolate to longer texts than it had seen during training.
• Practical consequence: It worked well for the first generation of models, but the assumption that the model would scale effortlessly to unknown lengths turned out to fall short in practice.

2. Learned Absolute Positional Encodings

Models such as GPT-2 and BERT switched to a simpler principle: let the model learn the position vectors itself during training. Position 1 gets its own vector that is adjusted via backpropagation, position 2 as well, up to the chosen maximum length (for example, 2,048 tokens).

• The idea: The network determines for itself what the most optimal representation is for each specific position in the sequence.
• Why it was devised: Learned parameters often performed slightly better and more stably on shorter sequences than the fixed mathematical formulas of sinusoidal encodings.
• Practical consequence: The maximum sequence length is hard-fixed during training. If a model is trained with 2,048 learned position embeddings, it simply has no vector for position 2,049. The model cannot work with longer texts without being retrained.

3. Relative and Rotary Encodings (RoPE and ALiBi)

Modern language models such as Llama and Mistral often use *Rotary Position Embedding* (RoPE) or *Attention with Linear Biases* (ALiBi). These methods move away from the idea of 'absolute' positions (such as "this is token 5") and instead focus on the 'relative' distance between tokens (such as "token A is 3 places before token B").

• The idea behind RoPE: Instead of adding a position vector to the embedding, the embedding vector is rotated within the mathematical space. The angle of rotation depends on the position of the token. The inner product between two rotated vectors therefore automatically contains information about the difference in position between those two tokens.
• Why it was devised: Relative distances are more representative of how language works. The relationship between an adjective and a noun is similar regardless of whether they appear at the beginning or in the middle of a page.
• Practical consequence: These methods deliver excellent performance and make it easier to extend models to very large context lengths using clever techniques (such as RoPE scaling).

Comparison of the techniques

Positional encoding and the maximum context length

The mechanism for positional encoding is directly linked to the context window of a language model. When a model is trained on a specific length, for example 4,096 tokens, it has learned how attention should be distributed across relationships within that distance.

If a text longer than the trained context length is simply fed into a model, performance loss occurs. This phenomenon is called the extrapolation problem. With learned positional encodings, the vectors for the higher positions are simply missing. With sinusoidal or rotary encodings, other problems arise: the model has to process rotation angles or frequencies it never encountered during training.

As a result, the attention mechanism goes off the rails. The attention scores become unpredictable, leading to incoherent output, repetition, or hallucinations. To stretch the context size of existing models without starting a full new training run, techniques are applied such as RoPE scaling or yarn. These techniques 'compress' or redistribute the position frequencies so that longer sequences fall within the model's familiar mathematical range.

What this means for you in practice

For users and developers working with language models, the way positional encoding works has concrete practical implications:

• Choosing models for long documents: If your application requires processing books, legal files, or large amounts of code, it's best to choose models that use modern relative positional encodings such as RoPE. These models retain their accuracy better over long distances.
• Understanding limits when prompting: Simply stretching the context window through software settings can lead to reduced precision if the model isn't optimized for this. This affects your strategy for context management.
• Understanding instruction order: Because relative positions determine how information is weighted, where instructions are placed in a prompt matters. Including important instructions at the beginning or the very end of a long prompt tends to produce better results in practice than placing them in the middle.

Positional encoding is an elegant example of how a fundamental limitation of a neural network — the lack of order in matrix computations — has been resolved in a structured way. It forms an essential building block that enables transformers to understand the intricate grammatical and semantic structures of human language.