Visual tokenizer simulator
When you enter text into a large language model, that text is never processed as a whole or simply as separate words. Instead, the model chops the input into smaller pieces that we call tokens. This interactive simulator lets you visually experience how different splitting methods work for both Dutch and English texts.
| Method | Number of pieces | Average number of characters |
|---|---|---|
| Per character | - | - |
| Per word | - | - |
| Approximate subwords | - | - |
Why language models work with pieces instead of whole words
Anyone hearing for the first time about how large language models work often wonders why the software doesn't simply look at complete words the way people do. A human reads the word 'language model' as one meaningful unit. A traditional computer program would look it up in a database of thousands of separate words. Both approaches, however, fall short for modern neural networks. If you treated every word as a unique category, the vocabulary would number in the millions of items, making the model gigantic and unworkable. Moreover, the model would completely get stuck on new words, spelling mistakes, or unusual contractions it had never encountered in the training data.
By choosing subwords or smaller character fragments, designers solve this fundamental problem. The model learns a fixed set of basic fragments, prefixes, suffixes, and common letter combinations. As soon as an unknown or brand-new word comes along, the model can still build that word from smaller, already known pieces. This provides enormous flexibility when processing language. Even if a term is completely new or comes from a rare field, the network can handle it without outright errors by falling back on the underlying building blocks. This approach forms the basis of modern tokenization algorithms such as Byte-Pair Encoding and WordPiece.
The simulator above shows this principle directly in practice. When you switch the setting to the approximate subword method, you see that common pieces remain intact while complicated or long words are chopped into manageable chunks. This gives you direct insight into how the software looks under the hood at the text you enter. The visual aspect, with the alternating colored blocks, also makes it immediately clear where the boundaries of the pieces lie and how spaces and punctuation marks are factored into the overall process.
The difference between the Dutch and English languages
A crucial insight for anyone working with language models is that different languages do not relate equally to the number of pieces needed. English, due to the structure of the language, is often very compact for tokenizers. Many English words are short and occur extremely frequently in public training data, so they are stored in their entirety as a single unit in the model's dictionary. Dutch, on the other hand, has properties that generally cause identical concepts to break down into more pieces than their English equivalents.
The main culprit in Dutch is the tendency to glue words together into long compounds. Where English speaks of a 'customer service portal', Dutch often uses the compound term 'klantenserviceportal'. A tokenizer that has not been trained on that specific compound term will split the word into separate recognizable components such as 'klanten', 'service', and 'portal', or even further into smaller syllables. As a result, Dutch texts often yield significantly more pieces than an English text with exactly the same informational value or the same number of written words.
In addition, inflections, conjugations, and grammatical endings play a major role. Dutch uses countless suffixes to indicate plurals, diminutives, or verb conjugations. Because not every possible conjugation fits into a model's dictionary as a unique whole, these words are chopped up during processing. This has direct consequences both for the cost of an API call and for the maximum amount of text you can feed to the model at once. Anyone paying per piece immediately notices that projects in the Dutch language consume relatively more than the same projects in English.
Consequences for cost, context length, and performance
The way text is chopped into pieces has direct and far-reaching practical consequences for anyone working with language models, whether professionally or as a hobby. Commercial providers almost always charge based on the actual number of pieces going in and out via the API. Because Dutch on average yields more pieces per sentence than English, the costs for Dutch projects turn out proportionally higher when you look purely at the number of words entered. It is therefore important to be aware of this hidden multiplication factor when budgeting AI applications.
In addition to the financial side, context length plays an equally important role. Every model has a strict limit on the total number of pieces it can remember and process at once, the so-called context window. When you want to feed long documents, manuals, or books to a model, those documents fill up the model's memory faster the more pieces the text breaks into. With a heavily inflected language, the available space therefore fills up sooner than with a compact language. Anyone who wants to handle this matter smartly should study per-token pricing models and take this into account when designing the prompt.
The quality of the answers can also be subtly affected by the way text is chopped up. If a specialized technical term or a unique code is broken into too many small pieces, the model can have trouble correctly interpreting the context of that specific word. You also see this with numbers, URLs, and programming code, which are often chopped up unfavorably. To get a grip on these kinds of technical challenges, it helps to read about tokenization explained and discover how you can optimize the input for reliable results in various situations.
Where this simulator stops being reliable
It is essential to understand the limits of an educational simulator like this one. The tool on this page uses fixed, simplified programming logic to show how text can be split up. Real commercial language models, however, use advanced statistical algorithms trained on enormous amounts of text corpus. Their vocabularies, often consisting of tens of thousands to several hundred thousand unique tokens, are the direct result of intensive machine learning processes and contain specific priorities for languages, programming languages, and emoji.
A real tokenizer does not look at a hardcoded list of Dutch prefixes the way this simulator does, but calculates, based on probabilities, which letter combinations most efficiently form a token together. This means the exact number of tokens for a given sentence can differ per model; one company's tokenizer leads to a completely different number than that of a competing platform. If you want to know exactly how many tokens your specific prompt consumes with a particular provider, you cannot rely on a general estimate or manual calculation.
For precise measurements and to avoid unexpected costs or error messages, you should always use the official development tools or libraries that the relevant provider itself makes available. Anyone who wants to dive deeper into how these different worlds come together will find useful background information in articles about multimodal tokenization explained and normalizing usage via token usage normalization across providers. The topic of embeddings explained also offers valuable context on how these pieces are ultimately converted into mathematical vectors.
