Building RAG properly: chunking, evaluation, reranking

You built a RAG chatbot. It works on the five questions you tried. Then a colleague asks something slightly different and it confidently answers from the wrong page.

This is almost never a generation problem. The model answered faithfully from what it was given — it was given the wrong context. RAG quality is retrieval quality, and retrieval is measurable. This article is about measuring it.

The plan:

1. Build a golden set so you have ground truth.
2. Define retrieval metrics that mean something.
3. Run chunking as an experiment, not a guess.
4. Add rerankingA second, slower model that re-scores the top candidates from the fast vector search and check whether it actually paid for itself.

Step 0 — Stop tuning blind

The usual RAG loop looks like this: change chunk size, ask your favourite question, squint at the answer, keep the change if it "seems better". That loop cannot detect a 5% regression, and it optimises for the three questions you happen to remember.

Replace it with a fixed evaluation you can run in thirty seconds.

Step 1 — Build a golden set

A golden set is a list of questions paired with the document IDs that genuinely contain the answer.

# golden.py
GOLDEN = [
    {
        "q": "What is the refund window for annual plans?",
        "relevant": ["billing.md#refunds"],
    },
    {
        "q": "Which regions is EU data residency available in?",
        "relevant": ["compliance.md#residency", "regions.md#eu"],
    },
    # ... 50 to 100 of these
]

Rules that matter more than the size of the set:

• 50 good questions beat 500 sloppy ones. You are going to read the failures by hand.
• Label documents or heading anchors, not chunk IDs. Chunk IDs change every time you re-chunk; anchors survive the sweep you are about to run.
• Include the hard cases you actually see: multi-hop questions, questions whose wording never appears in the source, near-duplicate documents, and questions that have no answer in the corpus.
• Write the questions before you look at retrieval output. Otherwise you write questions your system already answers.

The unanswerable questions are the ones people skip and then regret. Without them you cannot measure whether the system knows when to say "I don't know."

Step 2 — Metrics that mean something

Three numbers cover most of what you need.

Recall@k is the one to start with. If the correct chunk is not in the top k, no amount of prompt engineering saves the answer. Everything downstream is capped by it.

def recall_at_k(results, relevant, k):
    """results: ranked doc ids. relevant: set of correct doc ids."""
    top = set(results[:k])
    return len(top & set(relevant)) / len(relevant)


def reciprocal_rank(results, relevant):
    for i, doc in enumerate(results, start=1):
        if doc in relevant:
            return 1.0 / i
    return 0.0


def evaluate(retriever, golden, k=10):
    recalls, rrs = [], []
    for item in golden:
        hits = retriever(item["q"], k=k)
        recalls.append(recall_at_k(hits, item["relevant"], k))
        rrs.append(reciprocal_rank(hits, item["relevant"]))
    return {
        "recall@k": sum(recalls) / len(recalls),
        "mrr": sum(rrs) / len(rrs),
    }

A useful habit: report recall@5, recall@10 and recall@20 together. The gap between them tells you exactly what a reranker can buy.

recall@5   0.61  ############........
recall@10  0.79  ################....
recall@20  0.94  ###################.
                 ^^^^^^^^^^^^^^^^^^^^
                 reranker headroom = 0.94 - 0.61 = 0.33

If recall@20 is 0.94 and recall@5 is 0.61, a reranker has 33 points to recover by reordering. If recall@20 is 0.65, reranking is the wrong fix — your chunking or your embedding is losing the document entirely, and no reordering of the top 20 can conjure it back.

Step 3 — Chunking, run as an experiment

Chunking is where most of the easy wins are, and it is the parameter people set once and never revisit.

The tension is simple. Small chunks are precise — the embedding describes one idea, so similarity search is sharp — but they lose the context that makes the idea usable. Large chunks carry context, but their embedding is an average of several ideas, which makes it a weak match for any specific question.

The four strategies worth testing

# 1. Fixed-size by tokens
def fixed_chunks(text, size=512, overlap=64):
    toks = tokenize(text)
    step = size - overlap
    return [detokenize(toks[i:i + size]) for i in range(0, len(toks), step)]

Fast, dumb, and a perfectly respectable baseline. It cuts sentences in half, which matters less than people assume once you have overlap.

# 2. Recursive / structural — split on the biggest natural boundary that fits
SEPARATORS = ["\n## ", "\n### ", "\n\n", "\n", ". ", " "]


def recursive_chunks(text, size=512, seps=SEPARATORS):
    if len(tokenize(text)) <= size:
        return [text]
    for sep in seps:
        parts = text.split(sep)
        if len(parts) == 1:
            continue
        out, buf = [], ""
        for p in parts:
            candidate = buf + sep + p if buf else p
            if len(tokenize(candidate)) > size:
                out.extend(recursive_chunks(buf, size, seps[1:]))
                buf = p
            else:
                buf = candidate
        if buf:
            out.extend(recursive_chunks(buf, size, seps[1:]))
        return out
    return fixed_chunks(text, size)

For documentation, changelogs and anything with headings, this is usually the winner. It respects the structure the author already imposed.

3. Semantic chunking — embed each sentence, start a new chunk when consecutive sentences drift apart:

import numpy as np


def semantic_chunks(sentences, embed, threshold=0.82, max_tokens=512):
    vecs = embed(sentences)                    # (n, d), L2-normalised
    sims = (vecs[:-1] * vecs[1:]).sum(axis=1)  # cosine between neighbours

    chunks, buf = [], [sentences[0]]
    for sent, sim in zip(sentences[1:], sims):
        too_long = len(tokenize(" ".join(buf + [sent]))) > max_tokens
        if sim < threshold or too_long:
            chunks.append(" ".join(buf))
            buf = [sent]
        else:
            buf.append(sent)
    chunks.append(" ".join(buf))
    return chunks

Semantic chunking is the strategy that sounds best and disappoints most often. It costs an embedding call per sentence at index time, the threshold is corpus-specific, and on well-structured documents it usually loses to recursive splitting — because headings already encode the topic boundaries it is trying to rediscover. Test it; do not assume it.

4. Small-to-big — the one people underuse. Embed small chunks for precision, but return their larger parent for context:

# index time
for doc in docs:
    for parent in recursive_chunks(doc.text, size=2048):
        pid = store_parent(parent)
        for child in fixed_chunks(parent, size=256, overlap=32):
            index(embed(child), metadata={"parent_id": pid})


# query time
def retrieve(query, k=5):
    children = vector_search(embed(query), k=k * 3)
    parent_ids = dedupe([c.metadata["parent_id"] for c in children])
    return [load_parent(p) for p in parent_ids[:k]]

You search with a precise 256-token vector and hand the model a coherent 2048-token passage. In most evaluations this beats any single chunk size, because it stops forcing one number to serve two conflicting jobs.

Run the sweep

import itertools, json

STRATEGIES = {
    "fixed": lambda d, s, o: fixed_chunks(d, s, o),
    "recursive": lambda d, s, o: recursive_chunks(d, s),
    "small_to_big": lambda d, s, o: small_to_big(d, child=s // 4, parent=s),
}

rows = []
for name, fn in STRATEGIES.items():
    for size, overlap in itertools.product([256, 512, 1024], [0, 64, 128]):
        index = build_index(corpus, lambda d: fn(d, size, overlap))
        scores = evaluate(index.search, GOLDEN, k=10)
        rows.append({"strategy": name, "size": size,
                     "overlap": overlap, **scores})

print(json.dumps(sorted(rows, key=lambda r: -r["recall@k"])[:5], indent=2))

Twenty-seven index builds sounds heavy. On a 5,000-document corpus with a local embedding model it is a coffee break, and it replaces six weeks of arguing.

What tends to fall out of that sweep:

That last one deserves emphasis, because it costs one line:

chunk_text = f"{doc.title} > {heading_path}\n\n{raw_chunk}"

Two chunks that both say "Set this value to 30 seconds" are indistinguishable to an embedding model. Prefixed with Billing > Webhook retries and Search > Query timeouts, they are not.

Step 4 — Hybrid search, before you reach for anything clever

Dense vectors are bad at exact tokens: error codes, SKUs, function names, version numbers, surnames. BM25 is excellent at exactly those and hopeless at paraphrase. Run both and fuse the rankings.

Reciprocal Rank FusionA way to merge two ranked lists using only rank positions, so the two scoring scales never need to be normalised is the simplest thing that works:

def rrf(rankings, k=60):
    """rankings: list of ranked doc-id lists. Returns one fused ranking."""
    scores = {}
    for ranking in rankings:
        for rank, doc in enumerate(ranking, start=1):
            scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)


fused = rrf([bm25_search(q, k=50), vector_search(embed(q), k=50)])

No score normalisation, no weight to tune, one constant almost nobody changes. In practice hybrid retrieval is the highest-return change after chunking, and it is fifteen lines.

Step 5 — Reranking, and whether it earns its latency

A cross-encoderA model that reads the query and the document together in one pass and outputs a relevance score, instead of embedding them separately scores query and document jointly. That joint attention is why it is far more accurate than cosine similarity between two independently produced vectors — and also why it cannot be precomputed or indexed. You can only afford to run it on a shortlist.

query --> BM25 (top 50) ---+
                           +--> RRF (top 50) --> cross-encoder --> top 5 --> LLM
query --> vector (top 50) -+                     (rescore all 50)
          fast, approximate                      slow, accurate

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")


def retrieve_reranked(query, k=5, candidates=50):
    shortlist = rrf([
        bm25_search(query, k=candidates),
        vector_search(embed(query), k=candidates),
    ])[:candidates]

    docs = [load(d) for d in shortlist]
    scores = reranker.predict([(query, d.text) for d in docs])
    ranked = [d.id for _, d in sorted(zip(scores, docs), key=lambda x: -x[0])]
    return ranked[:k]

Now measure, because reranking is not free:

Those are illustrative shapes, not your numbers — but the shape is consistent. Reranking buys a large accuracy jump for a large latency cost, and doubling the candidate pool buys very little for double the cost.

Decisions that follow:

• Interactive chat: rerank 30–50 candidates. The latency is visible but tolerable behind a streaming response.
• Batch or offline pipelines: rerank 100+. Latency is free.
• Autocomplete-style budgets: skip the cross-encoder entirely; spend the effort on hybrid search and chunking.

Step 6 — Only now, evaluate the answer

With retrieval fixed, generation problems become visible. Two checks catch most of them: faithfulness (is every claim supported by the retrieved context?) and answer relevance (does it address the question?).

import anthropic, json

client = anthropic.Anthropic()

JUDGE = """You are grading a RAG answer.

Context given to the model:
<context>{context}</context>

Question: {question}
Answer: {answer}

Return JSON only:
{{"faithful": true/false, "unsupported_claims": [...], "addresses_question": true/false}}"""


def judge(question, context, answer):
    resp = client.messages.create(
        model="claude-opus-5",
        max_tokens=2000,
        thinking={"type": "adaptive"},
        messages=[{
            "role": "user",
            "content": JUDGE.format(
                context=context, question=question, answer=answer
            ),
        }],
    )
    text = "".join(b.text for b in resp.content if b.type == "text")
    return json.loads(text)

An unfaithful answer with correct retrieval is a prompting problem. A faithful answer that is wrong is a retrieval problem — and you now have the numbers to find it.

The order of operations

If you take one thing from this article, take the order. Teams routinely start at step 5.

1. Golden set. Nothing below this line is measurable without it.
2. Chunking sweep. Biggest win, lowest cost, often 10–25 recall points.
3. Title and heading prefixes. One line, consistently positive.
4. Hybrid search. Fifteen lines, large win on any corpus containing identifiers.
5. Reranking. Large win, real latency cost, only after 2–4.
6. Generation prompt and faithfulness checks. Last, because until here you were debugging the wrong stage.

Frequently asked questions

What chunk size should I use for RAG?

There is no universal answer, which is why the sweep exists. As a starting point: 512 tokens with 64 overlap using recursive splitting, or small-to-big with 256-token children and 2048-token parents. Then measure on your own corpus.

Is semantic chunking worth it?

Sometimes, and less often than its reputation suggests. On documents with headings, recursive splitting usually matches or beats it at a fraction of the indexing cost. Test it as one row in the sweep rather than adopting it on principle.

How many questions does a golden set need?

50 to 100 hand-written questions is enough to detect meaningful regressions and small enough that you will actually read the failures. Include unanswerable questions — they are the only way to measure whether the system abstains correctly.

Should I always add a reranker?

Only if recall@20 is meaningfully higher than recall@5. That gap is the maximum a reranker can recover. If both are low, fix chunking and retrieval first — a reranker cannot promote a document that was never retrieved.

Why is my RAG system confidently wrong?

Almost always because the wrong chunks were retrieved and the model answered them faithfully. Log the retrieved context for every bad answer; the failure is usually obvious the moment you look at what the model was actually given.

Next reading on Sythra Articles

• Fine-tuning vs RAG: how to actually choose
• Vector databases under the hood
• Embeddings explained mathematically