From Text to Numbers: Vector Embeddings in Practice

Computers and software algorithms fundamentally work with numbers, arithmetic operations, and mathematical matrix transformations. Humans, by contrast, communicate through language: dynamic, rich in nuance, synonyms, colloquialisms, and context-dependent meanings. To enable large language models (LLMs) and modern search systems to work effectively with written text, that text must be translated into a format the computer understands without losing its substantive meaning. That is exactly the role of vector embeddings.

This article explains how text is converted into numerical vectors, how these vectors capture meaning in a multidimensional space, how the similarity between different texts is calculated mathematically, and where this technology is applied in practice.

What are embeddings and why are they necessary?

In classical computer science, text was mainly analyzed based on exact characters (string matching). A search for the word "bicycle" only returned results in which the sequence of letters b-i-c-y-c-l-e appeared. Documents with synonyms such as "two-wheeler" or related terms such as "pedals" were overlooked by traditional search systems, unless extensive synonym lists were maintained manually.

The limitation of traditional keyword systems

Traditional search engines and databases use techniques such as TF-IDF (Term Frequency-Inverse Document Frequency) or BM25. These methods assign weights to individual words. Although this is effective for finding specific terms or unique names, they show two major limitations:

• No understanding of synonyms: The word "physician" and the word "doctor" are treated as two completely separate entities.
• Sensitivity to context: The word "bank" in "walking along the river bank" has an identical representation to "borrowing money from the bank", despite the completely different meaning.

Vector spaces and semantic meaning

An embedding is a numerical representation of a piece of text (a word, a sentence, or an entire document) in the form of a list of numbers, known as a vector. The crucial characteristic of modern embeddings is that they encode semantic meaning .

When texts are converted into vectors, they are placed in an imaginary multidimensional space. Texts that are similar in meaning end up close together in this space, regardless of the exact wording used. The well-known classic example from natural language processing (NLP) illustrates this principle:

Vector("Koning") - Vector("Man") + Vector("Vrouw") ≈ Vector("Koningin")

This mathematical relationship shows that the vector space understands human concepts and relationships and can express them numerically.

How does an embedding model work?

An embedding model is a specialized neural network that has been pre-trained on enormous amounts of text data. During this training, the model learns to recognize patterns and relationships between words and sentences based on the context in which they appear.

Dimensions and vectors explained

An embedding consists of a series of floating-point numbers (floats). The number of values in the series is called the dimension of the embedding. Common embedding models use vectors of, for example, 384, 768, 1536, or even 3072 dimensions.

Each dimension within a vector represents an abstract property or a semantic feature that the model has learned. A simplified representation of a 4-dimensional vector for three words might look as follows:

In reality, the dimensions in modern models cannot be directly mapped to a single human concept, but instead represent complex, combined patterns formed by the network.

Context and tokenization

Before a text is converted into an embedding, it is split into tokens. A token can be a whole word, but also part of a word or a punctuation mark. The embedding model then processes the sequence of tokens and generates, via so-called attention mechanisms (known from the Transformer architecture) a vector that takes into account the full context of the sentence.

For a deeper understanding of how the underlying Transformer architecture works, see the guide on the basic principles of the Transformer architecture is worth consulting.

Calculating similarity: Cosine Similarity

Once texts have been transformed into vectors, the substantive similarity between two documents, or between a search query and a document, can be determined mathematically. The most commonly used metric for this is cosine similarity (cosine similarity).

Cosine similarity measures the angle between two vectors in the multidimensional space, regardless of their length. The value always lies between -1 and 1 (or between 0 and 1 for normalized embeddings):

• 1.0: The vectors point in exactly the same direction; the texts are semantically identical or very strongly related.
• 0.0: The vectors are perpendicular to each other; there is no demonstrable semantic relationship.
• -1.0: The vectors point in opposite directions (rarely occurs with normalized text embeddings).

Practical applications in modern AI design

Vector embeddings form the foundation of virtually all modern search and analysis applications where natural language plays a central role.

Retrieval-Augmented Generation (RAG)

In Retrieval-Augmented Generation (RAG) for beginners embeddings are used to search external documents before an LLM formulates an answer. The data flow proceeds as follows:

1. Documents from a knowledge base are split into smaller text blocks (chunks).
2. Each text block is converted into a vector via an embedding model and stored in a database.
3. When a user asks a question, that question is converted into a vector using the same embedding model.
4. The vector of the question is compared to the stored vectors to retrieve the most relevant document fragments.
5. The relevant fragments are provided to the LLM together with the question as context.

To efficiently store and search millions of vectors, specialized infrastructure is used. For an overview of the most popular solutions, read the comparison of vector databases on directory.llmnet.nl.

Semantic search and categorization

Classic search systems fail when users search using different terms than those found in the source documents. With embeddings, a user who searches for "how do I fix a flat tire" also successfully finds documents titled "Instructions for bicycle repair and tire patching" You can find more about building such a system in the guide on semantic search on api.llmnet.nl.

Points of attention and challenges in implementation

Although embeddings are powerful, there are several important factors to consider in practical implementation:

• Chunking strategy: If a document is too large, the generated vector loses its specific details because all information gets averaged out. A well-considered splitting of text blocks is essential.
• Model choice and dimension size: Larger dimensions (e.g., 1536 compared to 384) offer higher accuracy but require significantly more memory and computing power when comparing large amounts of data.
• Multilingualism: Not every embedding model performs equally well in Dutch. For Dutch-language applications, it is necessary to test whether the model has sufficient language sensitivity for the specific domain terms.
• Model consistency: A vector generated by Model A cannot be directly compared to a vector from Model B. If an embedding model is updated, the entire database must be re-indexed.

Conclusion

Vector embeddings form the indispensable bridge between human language and the computing power of AI systems. By transforming text into a multidimensional number space, semantic relationships, synonyms, and contextual meanings become measurable. Whether it concerns building an advanced search function, a RAG application, or an automatic categorization system: understanding and correctly implementing embeddings is the key to a successful AI architecture.