Tokenization explained: how an LLM chops text into pieces

What is a token?

A token is the basic unit of data that an LLM processes. You can think of it as a piece of a word or a character string. Short, common words often make up exactly one token, while longer or complex words are split into multiple tokens by the tokenizer. After the text is split, each token is assigned a unique number (an ID). The LLM then performs its mathematical calculations using these numbers, not the text itself.

Why tokens matter: Costs and Context

If you build web applications and call AI models via an API, tokens are both your currency and your technical limit:

• Costs: API providers calculate prices per 1 million input and output tokens. Every time you send and receive data, it costs money. Especially with advanced systems like Retrieval-Augmented Generation (RAG), where you send large chunks of background information to the API, token counts — and therefore costs — quickly add up.
• Context Window: Each model has a maximum memory per API call, the so-called context window. A model with a window of 128,000 tokens can roughly remember hundreds of pages of text in a single session. As soon as you exceed this limit, the API will reject the prompt or information will have to be omitted.

Tokenization in Dutch: A hidden disadvantage?

Most tokenizers are primarily trained on massive amounts of English text. As a result, they are very efficient in English: 1 token is equivalent to about 0.75 words. Unfortunately, this is less favorable in Dutch. Our language contains many specific compound words and patterns that the tokenizer does not recognize as a single unit.

Example: The English word "apple" is 1 token. The Dutch word "gezelligheid" is split by many tokenizers into as many as 3 to 4 tokens (for example: "ge" - "zellig" - "heid"). This means that a Dutch prompt often consumes more tokens than an exact English translation, which directly impacts your API costs and the available space in your context window. Check out our token efficiency benchmark to see which LLMs perform most cost-effectively with the Dutch language.