What Is a Reranker? How Reranking Works in RAG Search

What is a reranker and what does it do in RAG search?

A reranker is a model that reads each candidate document from a first search together with the user's question, gives it a more precise relevance score, and reorders the list by that score. In RAG, it also makes sure the passages sent to the language model really match the question.

Think of a library. A helper quickly scans the shelves and puts twenty books on your desk. Then an expert reads the relevant pages of each book next to your question and moves the most useful ones to the top. The helper is fast and broad. But the expert is slow and precise.

The reranker is the expert. It does not run the search again. Instead, it only judges the candidates you already have, more carefully. This article covers this one term in depth, and we link to separate posts for the neighboring concepts.

What is two-stage retrieval?

Two-stage retrieval splits the work into two roles: a fast, broad first pass, then a slow, precise second pass. The first stage pulls a reasonable set of candidates out of a huge collection. The second stage looks only at that small set.

This split exists for a practical reason, because the alternatives do not scale. For example, running a precise model over the whole collection would be far too slow. Using only a fast method does not always put the best answer first. So you chain the strengths of both methods one after the other.

Here is the flow in short:

• The user asks a question.
• First, the retriever pulls a broad list of candidate passages from the collection.
• Then the reranker reads each candidate with the question and assigns a new score.
• Finally, the system sends the top few passages to the language model as context.

The Sentence Transformers documentation also describes this pattern as retrieve and re-rank. You will find the link in the sources section at the end.

What is a reranker compared with a bi-encoder and a cross-encoder?

A bi-encoder turns the question and the document into vectors separately, then compares the vectors. A cross-encoder reads the question and the document together in a single input. Most rerankers are cross-encoders, because they see both texts side by side and can judge how well they fit.

Also, the Sentence Transformers documentation describes the balance well. Bi-encoders are fast on large collections but can return irrelevant candidates. In contrast, cross-encoders judge relevance better, but they do separate work for every question and document pair, so they are expensive.

A simple way to remember the table: the bi-encoder answers "which ones look close?" The cross-encoder answers "which one really answers this question?"

How does a reranker work step by step?

A reranker feeds the question and a candidate text into the model as one input, and the model returns a relevance score. The score shows how well the candidate answers the question. The system repeats this for every candidate, then sorts the list by score.

Conceptually, the flow looks like this:

1. The retriever collects candidate passages for the question, for example the first few dozen.
2. Next, the system builds a pair of question and passage for each candidate.
3. Then the model reads each pair together and produces a score.
4. After that, the system sorts the candidates by score, highest first.
5. Finally, the few passages at the top go to the language model, and the rest drop out.

Nogueira and Cho showed one of the first strong examples of this idea in their paper "Passage Re-ranking with BERT." So this approach is not a passing trend. In other words, it is an established method in search research.

The absolute value of a score matters little. Instead, what matters is the order of candidates relative to each other. For that reason, do not compare raw scores from two different rerankers.

Another detail: because the model scores each candidate separately, you can run the work in parallel. Still, it has to read every candidate, so the total work grows with the candidate count. That is why the quality of the first stage directly affects the efficiency of the second.

Why do you need a reranker when the first search exists?

The first search usually returns passages that look close, but looking close is not the same as answering. For example, a passage that discusses the same topic without containing the answer can sit very near the question in vector space. Then the language model may trust it and build a wrong answer.

This is where the reranker helps. If the wrong passage enters the context, the model either fails to answer or fills the gap with its own guess. We cover that behavior in our post on AI hallucination.

A reranker makes a difference in these situations:

• The collection has many near-duplicate documents, such as version differences or similar policies.
• Questions are long, have several conditions, or are vague.
• The right passage is among the candidates but sits low in the list.
• You can send only a few passages to the language model.

The last point matters because the context window is limited. Also, every unnecessary passage takes space and can pull the model in the wrong direction.

What is hybrid search and how does it combine with a reranker?

Hybrid search combines keyword matching (the classic BM25 style of ranking) with vector search in one query. The keyword side is strong for exact phrases such as product codes or error numbers. However, the vector side catches the same meaning expressed in different words.

The Elastic and OpenSearch documentation both describe this combination in a vendor-neutral way. In both, the scores from the two searches are normalized and merged. A reranker then sits on top of the hybrid list as an extra layer.

In practice, the layers stack like this:

1. Keyword search produces a candidate list.
2. Vector search produces a second candidate list.
3. Then the system merges the two lists.
4. Finally, the reranker rescores the merged list together with the question.

According to the Elastic documentation, semantic reranking works the same way whether the first stage is keyword, semantic or hybrid. In other words, the reranker does not depend on the retrieval method. For the vector side, read our post on what embeddings are.

What does a reranker change in an example scenario?

Example scenario: a company's internal help assistant answers from the employee handbook. An employee asks, "how does carrying over annual leave work?" The retriever finds several leave-related sections: leave types, leave requests, public holidays and leave carryover.

In the first search, the "leave requests" section can land on top, because it shares many words with the question. The section that holds the real answer, "leave carryover," may drop to fourth place. So if the assistant takes only the top three passages, it answers with the wrong information.

Then the reranker reads each section together with the question. It sees that "carryover" is the center of the question and moves the right section to the top. So the same language model, with the same documents, builds a better answer.

This is an example scenario, not a real customer result. Still, the structure shows a common problem in real projects: the right information exists in the system, but it ranks too low.

How many candidates should you give a reranker?

The number of candidates you pass to the reranker is the main dial between quality and speed. If you pass too few, the right passage may never enter the list. On the other hand, if you pass too many, every query gets slower and costs more. So there is no magic number.

A practical approach works like this. First, look at your own test questions and see how far down the retriever list the right passage usually sits. Make the candidate window wide enough to cover that range, then trim what you do not need.

The Elastic documentation also points the same way. Rerankers typically work on the top K results, because reranking a long list would be very costly. Treat the window size as an engineering decision.

Do not mix up two separate numbers:

• The number of candidates entering the reranker (the wide window).
• The number of passages going to the language model (the narrow window).

The first sets your chance of catching the right passage. Meanwhile, the second sets how clean the context is.

How do you balance cost and latency?

A reranker means one extra model call for every question. That call adds latency and compute cost. Yet a language model that writes a long, empty answer from the wrong context also costs money. You have to set the balance with your own measurements.

When you calculate, look at these items:

• How long the reranker takes per query.
• How total work grows as the candidate count grows.
• Your infrastructure: your own server or a provider's hosted service.
• The savings from a smaller context going to the language model.

Teams often forget the last item, so it deserves a closer look. If the reranker lets you send fewer but better passages, your input token usage can drop. That saving can offset part of the reranker's cost. However, do not assume it. Instead, measure it on your own traffic.

Think about total traffic, not a single query. For example, for an internal assistant with few daily questions, the extra call hardly matters. In contrast, for a busy customer bot, the same call becomes a real line item.

If latency matters in your product, such as live chat, start with a small candidate window. Meanwhile, for background jobs that nobody waits on, you can use a wider window. For current model sizes and prices, check the provider's official page.

How do you measure reranker quality?

You measure a reranker with a fixed set of questions, not by gut feeling. For each question, you mark the correct passage in advance. Then you compare the ranking before and after the reranker on the same questions.

Search teams use ranking-focused metrics for this. Some, for example, check whether the right passage appears among the first few results. Others show the average position of the right passage. Do not invent thresholds. Instead, look at the before and after difference in your own collection.

Measure the final answer too, not only the ranking, because a better ranking can still leave answer quality unchanged. For answer-level measurement, see our post on LLM evals.

Follow these steps when you build the measurement:

1. Collect a representative set of real user questions.
2. Mark the correct source passage for each question.
3. Record the results with the reranker switched off.
4. Run the same questions with the reranker switched on.
5. Study the difference by question type instead of looking only at the average.

How do a reranker, embeddings and chunking relate?

The three concepts sit at different points of the same pipeline. Chunking splits documents into passages, embeddings turn those passages into vectors, and the reranker reorders the retrieved passages. So the reranker is a quality layer near the end of the line.

That order also carries a warning: a reranker cannot rescue badly split documents. If an answer is cut across two passages, no ranking can make it whole again. For this reason, start with the principles in our post on what chunking is.

For the full picture, what is RAG is a good starting point. The reranker is only one link in that chain, so do not treat it as a stand-alone fix.

When you troubleshoot, check the pipeline from start to finish. First ask whether the documents are clean. Then check whether the passages make sense and whether the retriever puts the right candidate in the list. Last, see whether the reranker fixes the order.

What is a reranker's difference from similar terms?

People often confuse the reranker with neighboring terms. The table below puts each term's place and job side by side. That way, you can tell which tool solves which problem.

For the guardrail row, see our post on AI guardrails. Every row in the table has its own post, so we do not repeat them here.

What is a reranker's role in real use cases?

A reranker can add value anywhere people say, "there are many documents, but finding the right one is hard." The common thread is that the right result needs to be among the first few. The examples below are general scenarios, so please do not read them as real client cases.

• Corporate knowledge assistant: Picking the right section among policies, procedures and handbooks.
• Customer support bot: Putting the help article that truly solves the problem first among similar-looking articles.
• Site search: Moving the results closest to user intent to the top in product or content search.
• Document search: Finding the paragraph that best fits the question in contracts, reports and technical documents.

For instance, take site search on an online store. A search for "winter jacket" can return both jackets and accessories. A reranker reads the intent together with the query and can move the jackets up. Therefore the logic matches the document assistant.

What kinds of rerankers exist?

When you ask what is a reranker, you meet more than one type. The most common is the cross-encoder, which reads question and document together. However, some teams let a language model do the ranking, and others settle for simple rules and score blends. All of them live under the name reranking.

You can think of three groups:

• Purpose-trained models: Relatively small and fast models trained only for ranking.
• Hosted services: Ranking calls that providers offer through an API, with no setup.
• Language-model ranking: Asking a large model to read the candidates and pick the best one.

Each type balances latency, cost and privacy differently. For example, a hosted service gets you started fast, but your text goes to the provider. A model on your own infrastructure keeps data in-house, but the operating work is yours. Check current options in the providers' official documentation.

How does a reranker behave on non-English content?

Not every reranker works equally well in every language. Instead, a model ranks best in the languages it saw most during training. So a model that does well in English tests may not deliver the gain you expect on your German or Turkish documents. Do not accept that assumption without measuring.

For example, word forms change a lot in some languages. German compounds and Turkish suffixes can hide the same concept behind different surface forms. Keyword search can miss that. However, a good reranker may handle the variety better, but you have to confirm it with your own questions.

Add one more check for multilingual collections, because the question may be in one language and the document in another. Read the model's language support in its official documentation, then try it with real examples. A language-support claim and your own results are not the same thing.

How do you use reranker scores in the final answer?

The reranker's order does more than decide which passages enter the context. It also hints at how reliable the answer will be. If even the top passage scores low, the question probably has no match in the collection. In that case, saying "I could not find it" is more honest than inventing an answer.

Keep these points in mind when you design that behavior:

• For low-scoring results, hand the user to a person instead of answering.
• Show the source passage or document title next to the answer.
• Ask the user to clarify vague questions.
• Do not copy a threshold from somewhere else. Instead, set it with your own data.

This way the reranker is more than a sorting tool. It also helps the system know its own limits. An assistant that can say "I don't know" in the right place is more trustworthy than one that invents an answer to every question.

Does a reranker change anything for security?

A reranker is not a security layer, but it affects which passages reach the context. That puts it inside the security discussion. For example, a document that carries a malicious instruction can reach the language model if it scores high. The ranking model does not judge whether an instruction is harmful. Instead, it scores only relevance to the question.

For that reason, do not use ranking as a replacement for security checks. Verify document sources, check outputs and limit the context that goes to the model. For the attack type itself, see our post on prompt injection. We therefore do not go into detail here.

Also, for documents with personal data, apply access rights before ranking. A document the user is not allowed to see should never enter the candidate list. This is not legal advice, so talk to your own specialist about data protection duties.

Do you need a reranker in every project?

No, you do not need one in every project. For example, for a small, clean collection with short, clear questions, the first search is often enough. In that case, adding a reranker only adds complexity. The healthiest path is to build the simple pipeline first and measure it. This answers what is a reranker's real value for your case: you see the gap before you pay for the fix.

These signs tell you to consider a reranker:

• The right passage exists in the system but often does not reach the answers.
• Users complain, "it is in the document but the assistant could not find it."
• Answer quality drops as the collection grows.
• You see confusion between documents that look very much alike.

If none of these apply, improve the other links first. Chunking, document cleanup or query rewriting often brings a cheaper win. For example, old and current versions mixed in one collection are a data problem that a reranker cannot solve.

If you are building an assistant over company documents, our RAG development page explains how we approach such projects.

What limits and risks come with a reranker?

A reranker is a ranking layer, not a miracle. Its biggest limit is this: if the right passage is not among the candidates, the reranker cannot invent it. When the retriever never lists the right document, the reranker only picks the least wrong one.

Other risks include:

• Latency: Every query gets one more model call, so response time grows.
• Domain mismatch: A general-purpose reranker may be weak in a very specialized industry vocabulary.
• False confidence: A high score does not always mean a correct answer.
• Maintenance load: Ranking behavior can shift when the model changes, so you need to measure again.
• Data privacy: If you use an outside hosted service, you must be clear about where the text goes.

Is a reranker the same as ranking with a language model?

No, but they are related. Some teams, in fact, let a large language model do the ranking directly. The model reads the candidates and picks the best fit. This approach is flexible, but it is usually slower and more expensive. A reranker, by contrast, is often a smaller model trained for this one job.

Think about it this way: if you need ranking to be very fast, a purpose-built reranker fits. If your ranking rules are complex and different every time, language-model ranking may be worth testing. So do not decide before you compare both on your own test set.

When you compare, look at consistency as well as quality. A model that gives the same order for the same question every time makes testing and debugging easier. Also, we do not push any specific product here. Model names and versions change fast, so check the providers' official documents and test with your own data.

Where should you start when adding a reranker?

To add a reranker, first measure the current system. Then switch the reranker on in a test environment with a small candidate window and compare the difference. This order protects you from needless cost, so do not push it straight to production.

Here are practical starting steps:

1. Prepare a test set from real questions.
2. Record the ranking quality of the current pipeline.
3. Choose a reranker, either open source or a hosted service.
4. Start the trial with a small candidate window.
5. Report latency and quality together.
6. If the difference is meaningful, roll it out to production in stages.

At Talha Aslan and team, we suggest basing these decisions on measurement. That is because the same method can make a big difference in one collection and change nothing in another.

What mistakes do teams make when setting up a reranker?

The most common mistake is adding a reranker without measuring. Next comes trying to cover a retriever problem with a reranker. Finally, teams grow the candidate window needlessly and leave latency unchecked. Fortunately, all of these are avoidable.

Use this list as a warning:

• Deciding "it feels better" without test questions.
• Ignoring chunking and document cleanup problems.
• Reranking only the first few candidates instead of a wide enough window.
• Treating a high score as absolute truth.
• Skipping the new measurement after a model change.
• Not showing the user the source behind the answer.

Do not underestimate the last point. For instance, showing which document the answer rests on lets users verify it, and it helps you find the problem when something goes wrong.

How do you explain what a reranker is to your team?

When you explain what a reranker is to a non-technical manager, skip the detail. "The first search collects candidates, the reranker picks the best one" is often enough. Then use the library comparison from the start of this article, because an analogy makes the concept easier to remember.

Give these three messages clearly in the meeting:

• A reranker does not find new information, it fixes the order of existing candidates.
• It can raise quality, but it also brings latency and cost.
• The decision should rest on measurement, not on assumption.

For the developer team, offer a different frame: which candidate window, which model, which metric. Before the decision meeting, make sure everyone looks at the same test set. That way, the debate moves from opinion to data.

What is a reranker in short, and what is the final checklist?

In short, a reranker is a quality layer that reads the first search's candidates with the question and reorders them more precisely. It separates fast retrieval from precise ranking. As a result, the context that reaches the language model is cleaner.

Review this list before you decide:

• Is the right passage among the candidates, or does the retriever never find it?
• Are your test questions and marked correct sources ready?
• Is your latency and cost budget clear?
• Are chunking and document cleanup good enough?
• Have you tried hybrid search?
• Do you measure answer quality separately?

For a wider frame, our post on context engineering will also help. The reranker is, after all, only one of the technical ways to select the right context.

Sources

The conceptual explanations in this article rest on the primary and official sources below. For current details, go to these pages directly:

• Nogueira and Cho, Passage Re-ranking with BERT (arXiv abstract).
• Sentence Transformers official documentation, Retrieve and Re-Rank.
• Elastic official documentation, Semantic reranking.
• OpenSearch official documentation, Getting started with semantic and hybrid search.