Re-ranking explained: cross-encoders versus vector similarity

In a standard Retrieval-Augmented Generation (RAG) architecture, retrieving relevant context is the first and often most fragile step. The standard approach relies almost exclusively on so-called bi-encoders: the query and document chunks are converted independently into dense vectors, after which cosine similarity or a dot product determines the ranking. While this process can search millions of vectors in milliseconds, it frequently produces a ranked list that misses crucial semantic nuances. Re-ranking using cross-encoders resolves this shortcoming by implementing a two-stage pipeline: a fast, coarse pre-selection followed by a rigorous re-evaluation.

In this article, we break down the mathematical and computational differences between bi-encoders and cross-encoders. We examine why pure vector similarity falls short with negations, factual constraints, and word order, how the full attention of cross-encoders successfully captures these interactions, and how to efficiently integrate these components into production systems without introducing unacceptable latency.

The fundamental limitation of bi-encoders and vector similarity

Bi-encoders operate according to a strictly decoupled architecture: text block A and text block B pass through the same (or an identically trained) transformer model and each yield a single fixed embedding vector of, for instance, 768, 1024, or 1536 dimensions. The final similarity score is nothing more than the geometric angle between those two static points in a high-dimensional vector space. This principle enables vector databases to filter top-k results within milliseconds using approximate nearest neighbor algorithms.

The inherent flaw in this technique is the so-called information bottleneck. An entire paragraph containing multiple facts, conditions, and caveats must be compressed into a single numerical anchor point. During this compression, the subtle interplay between individual words is lost. After all, self-attention occurs exclusively within the query and within the document chunk, but never between the query and the passage simultaneously.

This creates a blindness to asymmetric relationships. Take a query like "Which subsidies in 2026 do not apply to residential heat pumps?". A bi-encoder projects both the query and policy texts about heat pump subsidies into virtually the same region of vector space. The word "not" barely carries enough numerical weight in an average pooling vector to penalize documents about subsidies that *do* apply. The result: the top 5 contains texts that are semantically close in theme, but miss the exact logical answer.

How a cross-encoder works: joint attention mechanisms

A cross-encoder approaches the comparison problem in a fundamentally different way. Instead of computing two separate vectors and multiplying them, the query ($Q$) and the candidate document ($D$) are concatenated into a single input sequence, separated by a special delimiter token:

[CLS] Zoekvraag Q [SEP] Kandidaatdocument D [SEP]

This combined sequence is passed in its entirety through all transformer layers of the model. From the very first encoder layer, every token in the query can direct attention directly to every token in the candidate document via the self-attention mechanism, and vice versa. The output is not a vector representation, but a direct numerical relevance score (typically normalized via a logit or sigmoid) between 0 and 1.

Thanks to this cross-attention, the network can recognize complex patterns:

• Direct negation resolution: The model immediately detects whether a condition in the query conflicts with an exclusion clause in the text.
• Term matching in context: Technical terms that are spelled identically but differ by context (polysemy) are weighted correctly based on surrounding words.
• Syntactic dependencies: Who does what to whom? With bi-encoders, "Company X acquires Company Y" often yields the exact same cosine score as "Company Y acquires Company X". A cross-encoder immediately detects the difference in grammatical structure.

Mathematical and computational differences

Why don't we simply use cross-encoders for the entire database? The answer lies in computational complexity and the feasibility of pre-computing. With bi-encoders, document embeddings are computed offline during database indexing; at runtime, only the query needs to be embedded ($O(1)$ inferences), after which the vector index performs a fast comparison.

With a cross-encoder, pre-computing is by definition impossible. Because the calculation depends on the unique combination of both query and document, a full forward pass through the neural network must be performed for every candidate document. If you were to search a collection of 100,000 documents with a cross-encoder, 100,000 transformer evaluations would have to be run for a single query. This results in latencies of tens of seconds or even minutes.

The Two-Stage Retrieval Pipeline

The standard solution to combine the best of both worlds is a two-stage retrieval pipeline. In this approach, each stage has its own distinct goal: phase 1 maximizes recall (finding as many potential hits as possible), while phase 2 optimizes precision (placing the best documents at the top).

The process works as follows:

1. First Phase (Dense/Sparse Retrieval): The search query is fired at a vector database (such as Qdrant, Milvus, or pgvector) or a hybrid index (vectors combined with BM25). The index quickly retrieves a broad candidate list, for example the top-50 or top-100 documents. Those managing their own infrastructure can set up a local vector database to perform this first retrieval stage with minimal network latency.
2. Second Phase (Cross-Encoder Re-ranking): The raw list of 50 text snippets is passed along with the original search query to the cross-encoder model. This model computes an exact relevance score for all 50 pairs and re-sorts the list.
3. Truncation & Context Injection: Only the top-scoring 3 to 5 snippets are selected and placed into the final prompt for the generative language model.

This separation keeps the total response time within acceptable limits (typically between 30 and 150 milliseconds for the re-ranking step on a modest GPU or fast CPU), while dramatically improving the quality of the context.

Practical Comparison and Quality Metrics

To evaluate what adding a cross-encoder delivers, we look at standardized Information Retrieval metrics: MRR@k (Mean Reciprocal Rank) and NDCG@k (Normalized Discounted Cumulative Gain). While bi-encoders often achieve a solid recall@100 on benchmark datasets like BEIR, the NDCG@10 plummets when documents with superficial word overlaps accidentally rank higher than substantively correct paragraphs.

In production measurements, we see a consistent pattern: adding a re-ranker on top of a bi-encoder provides an average increase of 15% to 35% in MRR@10. Especially in domains with dense terminology (legal, financial, and medical), the re-ranker prevents documents that happen to repeat the same terms, but reach a different conclusion, from polluting the context.

In addition, document segmentation plays a crucial role. If text blocks in the database are split awkwardly, even the best re-ranker cannot recover the contextual coherence. For a solid implementation, explore the various chunking strategies to smartly split documents for RAG, ensuring snippets maintain the exact right length for the cross-encoder's attention span.

Concrete Dutch Practical Example: Searching Policy Documents

Let us look at a realistic case study within a Dutch municipality using a RAG system to answer questions about local regulations (APV).

The citizen's search query:
"Do I need a felling permit for a diseased birch with a trunk circumference of 45 cm on private property?"

The vector database contains three relevant fragments from the tree felling policy:

• Fragment A: "A felling permit is required at all times for felling trees on public municipal land, regardless of the trunk circumference or the health of the tree."
• Fragment B: "Private tree owners require a permit to fell woody vegetation with a trunk circumference of at least 50 cm, measured at a height of 1.30 meters above ground level."
• Fragment C: "Exemption clause: For demonstrably diseased trees on private property, the permit requirement is waived if the circumference is less than 60 cm and there is an imminent danger to the surrounding area."

What happens with pure bi-encoder vector similarity? Fragment A scores very high on terms such as "felling permit", "felling trees", and "regardless of health". Fragment B scores high on "private tree owners" and "trunk circumference". Due to its more abstract legal phrasing ("woody vegetation", "fell", "exemption clause"), Fragment C often has lower cosine similarity relative to the citizen's colloquial query. Chances are high that Fragments A and B emerge as the top-2 from the vector search, leading the language model to conclude that no permit is required based on the 50 cm rule, while missing the specific exemption rule for diseased trees (Fragment C).

If we place a cross-encoder downstream of the retrieval step, it analyzes the combined interaction. The model recognizes that the query simultaneously asks about "diseased birch" AND "private property" AND "45 cm". In Fragment C, the self-attention matrix immediately detects the correlation between "diseased trees", "private property", and the threshold value. Fragment C receives top priority (score 0.94), followed by B (score 0.71), while A drops significantly (score 0.18). The LLM receives the correct context and generates a flawless answer.

Popular re-ranking models and technological variants

Anyone implementing a re-ranking component today can choose from various open-source architectures and specialized services:

• BGE-Reranker (BAAI): Based on the RoBERTa and XLM-RoBERTa architecture (such as bge-reranker-v2-m3). Excellent multilingual performance, including robust support for Dutch, and compact enough to run locally on a GPU or a fast multi-core CPU.
• FlashRank: An ultra-lightweight re-ranker designed for CPU environments. Utilizes compressed and distilled MiniLM-like networks, enabling latencies below 10-20 ms without GPU hardware.
• ColBERT (Late Interaction): An interesting hybrid variant that is neither a pure bi-encoder nor a traditional cross-encoder. ColBERT preserves token-level embeddings for each document and computes cross-similarity at lightning speed during runtime via the MaxSim operator. This provides a fraction of the cross-encoder latency while retaining much of the precision.
• Specialized APIs (e.g., Cohere Rerank, Jina Reranker): Cloud-based models trained on massive volumes of search interactions that deliver excellent out-of-the-box performance without requiring you to host model weights yourself.

Hardware Impact, Throughput, and Production Strategies

Adding a cross-encoder to the request loop requires clear architectural choices. Where vector similarity relies purely on memory speed and simple vector calculations in C++/Rust, the re-ranker demands actual transformer compute (matrix multiplications).

Below is a minimalist Python example demonstrating how to use the library sentence-transformers to re-rank a list of candidate results from a vector search:

from sentence_transformers import CrossEncoder

# Laad een meertalig re-ranker model
model = CrossEncoder('BAAI/bge-reranker-v2-m3', max_length=512)

query = "Welke regels gelden voor zonnepanelen op een monument?"
kandidaat_documenten = [
  "Subsidie voor isolatie kan worden aangevraagd via het landelijk loket.",
  "Voor monumentale panden is altijd een omgevingsvergunning nodig bij zonnepanelen.",
  "Zonnepanelen op reguliere daken zijn vergunningsvrij mits binnen het dakvlak geplaatst."
]

# Bouw de paren op voor de forward pass
paren = [[query, doc] for doc in kandidaat_documenten]
scores = model.predict(paren)

# Koppel documenten aan hun berekende scores en sorteer aflopend
gerangschikt = sorted(
  zip(kandidaat_documenten, scores),
  key=lambda x: x[1],
  reverse=True
)

for doc, score in gerangschikt:
  print(f"[{score:.4f}] {doc}")

In a production environment with hundreds of concurrent users, keep the following rules of thumb in mind:

• Limit the top-k: Send a maximum of 25 to 50 candidates to the cross-encoder. Anything beyond that yields exponentially higher latency for marginal precision gains.
• Dynamic batching: Execute score calculations in batches across the GPU to make optimal use of Tensor Cores.
• Threshold pruning: Filter out documents with a relevance score below a hard threshold (e.g., < 0.35) immediately to avoid polluting the downstream model's context window. This aligns seamlessly with the principles of advanced context engineering to keep prompts concise and targeted.

Pitfalls, Limitations, and When Not to Use a Cross-Encoder

Although cross-encoders significantly enhance the response quality of RAG applications, they are not a silver bullet for every architectural challenge. There are specific scenarios where adding this layer actually works against you:

1. Strict real-time requirements (<50 ms total round trip): When an application must respond within a few dozen milliseconds (such as type-ahead suggestions or voice agents), the added latency of a cross-encoder is often unacceptable. In these scenarios, a well-tuned hybrid search index (dense + sparse BM25) with Reciprocal Rank Fusion (RRF) offers a faster alternative.

2. Limited context length of the re-ranker: Many popular cross-encoders have a context window capped at 512 tokens. If your document chunks exceed this limit, text gets truncated. Words at the end of a long fragment simply won't be factored into the cross-attention computation.

3. Domain-specific mismatch: A generic cross-encoder trained on general web text and Wikipedia won't automatically understand specialized source code, medical records, or cryptic log files. Without fine-tuning, an out-of-the-box model might erroneously rank relevant technical fragments lower than a standard keyword search would.

Leveraging bi-encoders for coarse filtering alongside cross-encoders for fine-grained re-ranking creates a robust, highly reliable retrieval foundation. This prevents language models from hallucinating over semi-relevant documents and allocates your hardware compute exactly where it delivers the highest quality impact.