Hybrid search for RAG: BM25 plus embeddings

Most RAG systems start with vector search and work beautifully on the demo questions. Then a user pastes an error code, a product SKU or a person's surname, and the right document is nowhere in the top results. Embeddings capture meaning; they are poor at exact strings. Classic keyword search has the opposite strengths. Hybrid search runs both and merges the results. It is one of the cheapest retrieval upgrades available.

Two retrievers, two blind spots

Neither wins everywhere. Real queries mix both kinds, so use both.

Part 1 — BM25 from scratch

BM25A ranking function that scores documents by how often query words appear, how rare those words are, and how long the document is is the standard keyword scorer behind most search engines. The idea in three parts:

• Term frequency: a document mentioning the query word more is more relevant, with diminishing returns
• Inverse document frequency (IDF): rare words matter more than common ones
• Length normalisation: a long document should not win just for containing more words

import math
import re
from collections import Counter


def tokenize(text):
    return re.findall(r"[a-z0-9]+", text.lower())


class BM25:
    def __init__(self, docs, k1=1.5, b=0.75):
        self.k1, self.b = k1, b
        self.tokens = [tokenize(d) for d in docs]
        self.lengths = [len(t) for t in self.tokens]
        self.avg_len = sum(self.lengths) / len(self.lengths)
        self.tf = [Counter(t) for t in self.tokens]

        df = Counter()
        for t in self.tokens:
            df.update(set(t))                      # count documents, not occurrences
        n = len(docs)
        self.idf = {
            w: math.log((n - c + 0.5) / (c + 0.5) + 1) for w, c in df.items()
        }

    def scores(self, query):
        result = []
        for i, tf in enumerate(self.tf):
            score = 0.0
            for word in tokenize(query):
                if word not in tf:
                    continue
                f = tf[word]
                norm = f + self.k1 * (1 - self.b + self.b * self.lengths[i] / self.avg_len)
                score += self.idf[word] * f * (self.k1 + 1) / norm
            result.append(score)
        return result

k1 controls how quickly repeated words stop helping; b controls how strongly length is penalised. The defaults (1.5 and 0.75) are sensible starting points.

Try it on a tiny corpus:

docs = [
    "Error code E1042 means the payment gateway rejected the card.",
    "How to refund a customer when a payment fails.",
    "Our returns policy lets you send items back within 30 days.",
    "Troubleshooting login problems and password resets.",
]

bm25 = BM25(docs)
print([round(s, 2) for s in bm25.scores("E1042")])
# [1.15, 0.0, 0.0, 0.0]   exact code found at once

print(bm25.scores("reimbursement"))
# [0.0, 0.0, 0.0, 0.0]    no document contains the word

The exact code lands precisely on the right document. But a query using a word the corpus never uses — "reimbursement" when the docs say "refund" — scores zero everywhere. That is the vocabulary mismatch problem, and it is what embeddings solve.

(For production, use a maintained library such as rank_bm25, or the built-in full-text features of your search engine or database. Writing it yourself once is for understanding.)

Part 2 — Vector search

Embeddings map text to vectors so that similar meanings sit close together. Retrieval is then a nearest-neighbour lookup (see how vector databases work).

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

doc_vectors = model.encode(docs, normalize_embeddings=True)


def dense_scores(query):
    q = model.encode([query], normalize_embeddings=True)[0]
    return doc_vectors @ q          # cosine similarity, since vectors are normalised

Any embedding provider works the same way: embed the documents once, embed the query at search time, take dot products. Now "reimbursement" or "get my money back" should rank the refund document first, even with zero shared words.

Part 3 — Merge the rankings

The two scorers speak different languages: BM25 scores are unbounded (1.15, 12.7, ...) and cosine similarities sit between -1 and 1. You cannot just add them. Normalising scores works but is brittle. The robust trick is reciprocal rank fusionCombining rankings by summing 1/(k + rank) for each document across the lists; ignores raw scores entirely (RRF): ignore scores, use only ranks.

RRF score(doc) = sum over rankers of  1 / (k + rank)

with k = 60 as the conventional constant (it dampens the advantage of being first).

def ranks_from_scores(scores):
    order = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)
    return order           # list of doc ids, best first


def rrf(rankings, k=60):
    fused = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(fused, key=fused.get, reverse=True)


def hybrid_search(query, top_k=3):
    bm = ranks_from_scores(bm25.scores(query))
    dense = ranks_from_scores(dense_scores(query))
    return rrf([bm, dense])[:top_k]

Worked example

Suppose a document is ranked 1st by BM25 and 3rd by vector search:

1/(60+1) + 1/(60+3) = 0.01639 + 0.01587 = 0.03226

A document ranked 2nd by both gets 2/62 = 0.03226 — practically tied. A document ranked 1st by only one ranker and absent from the other's top results gets about 0.0164. So RRF rewards agreement: documents both retrievers like rise to the top, while a document only one retriever loves still has a chance if it is ranked very high.

One practical detail: rank only the top 20–50 candidates from each retriever rather than the full list, and let documents missing from one list contribute nothing from that list.

Part 4 — Rerank the merged list

Hybrid search improves recall: the right chunk is more likely to be somewhere in your candidate set. A reranker then improves precision by reading the query and each candidate together and re-ordering them. A typical pipeline:

query ──► BM25 top 30 ┐
                      ├─► RRF merge ─► top 20 ─► reranker ─► top 4 ─► LLM
query ──► vectors top 30 ┘

This is the structure described in chunking, retrieval evaluation and reranking. Add stages only when measurement shows they help.

Measure it, don't assume it

Hybrid usually wins, but "usually" is not a result. Build a small set of real questions, each with the document that should answer it, and compare:

def recall_at_k(search_fn, cases, k=5):
    hits = 0
    for question, correct_doc in cases:
        if correct_doc in search_fn(question, top_k=k):
            hits += 1
    return hits / len(cases)

# compare: bm25 only, dense only, hybrid

Include exact-match queries (codes, names) and paraphrase queries in the set, since those are where the two methods diverge. If hybrid does not beat your best single method on your data, do not pay the complexity.

Many databases do this for you

You rarely need to hand-roll the merge in production. Search engines and some vector databases (Elasticsearch/OpenSearch, Postgres with full-text plus pgvector, and others) offer hybrid queries with built-in fusion. The concepts above tell you what to configure and what to measure.

Tuning tips

• Tokenisation matters for BM25. Keep identifiers like E1042 and v2.3 intact, and consider stemming for prose.
• Chunk size affects both retrievers. Keep chunks focused; see the chunking guide.
• Weighted fusion is possible (multiply one ranker's contribution by a weight) if your queries skew heavily exact-match or heavily semantic.
• Embed the same text you index for BM25, or retrieval results will disagree for confusing reasons.

FAQ

What is hybrid search?

Hybrid search combines keyword search (like BM25) with vector similarity search and merges their results, so both exact terms and semantic meaning influence the final ranking.

What is BM25 used for?

BM25 scores how well documents match a keyword query, based on term frequency, word rarity and document length. It powers keyword search in many search engines.

What is reciprocal rank fusion?

RRF merges several ranked lists by giving each document 1/(k + rank) from every list and summing. It ignores raw scores, so it works even when rankers use incompatible scales.

Is hybrid search better than vector search alone?

Often, especially for queries with exact identifiers, names or rare terms. Test both on your own questions to confirm before adding the complexity.

Next reading on Sythra Articles

• RAG chunking, retrieval evaluation and reranking
• Vector databases under the hood
• Build a chatbot that answers from your own PDFs