Fine-tuning vs RAG: how to actually choose

The question arrives in the same shape every time: "Should we fine-tune a model on our documentation, or use RAG?"

The question carries a hidden assumption — that these are two ways to do the same thing. They are not. Fine-tuning and RAG solve different problems, and the reason "should I fine-tune on my docs?" is usually the wrong question is that fine-tuning is bad at teaching facts and RAG is bad at teaching behaviour.

The one-line version:

RAG changes what the model knows. Fine-tuning changes how the model behaves.

Everything below is that sentence with evidence attached.

Why fine-tuning is bad at facts

Fine-tuning adjusts weights to make certain outputs more likely. It does not build a lookup table. When you fine-tune on a document set, the model does not learn "the refund window is 30 days" as a retrievable fact — it learns a slightly stronger association between refund-shaped contexts and 30-day-shaped tokens.

Three consequences follow, and each bites in production.

It cannot be updated cheaply. A fact changes; you retrain. RAG updates by writing one row.

It cannot cite. The knowledge is smeared across weights. There is no passage to link to. For anything auditable — legal, medical, financial, internal policy — this alone disqualifies it.

It hallucinates in the shape of your data. This is the genuinely dangerous failure mode. A model fine-tuned on your documentation learns your house style, your terminology and your formatting, and then produces confident, perfectly-styled answers about API endpoints that do not exist. It became fluent in your domain without becoming accurate in it. Base-model hallucinations often look wrong; fine-tuned hallucinations look exactly right.

What fine-tuning is genuinely good at

Behaviour. Format. Consistency. Compression.

Two of these deserve unpacking.

Distillation is the most underrated reason to fine-tune. Take a frontier model, run it over 10,000 examples of your task, keep the outputs that pass review, and fine-tune a small model on them. You often reach 95% of the quality at a fraction of the per-call cost and latency. This is a cost optimisation, not a capability one — and it is the case where fine-tuning most reliably pays for itself.

Deep format compliance is the other. If you need 40-line structured clinical notes in a house format, no prompt is as consistent as 2,000 fine-tuning examples. You can spend 3,000 tokens of system prompt on format rules that are followed 92% of the time, or bake them into weights and follow them 99% of the time with a 200-token prompt.

The decision, as a flowchart

Is the model failing because it lacks INFORMATION?
|
+-- YES --> Does that information change over time?
|           +-- YES ---------------------------> RAG
|           +-- NO, stable and small -----------> put it in the prompt (+ cache it)
|
+-- NO, it lacks the right BEHAVIOUR
            |
            +-- Have you seriously tried prompting + few-shot?
            |   +-- NO --> do that first. Really.
            |
            +-- Is the behaviour expressible in under 20 rules?
            |   +-- YES --> prompt + structured outputs
            |
            +-- Is it style/format/consistency at scale,
                or a cost problem on a narrow task?
                +-- YES --> fine-tune (usually LoRA)

The uncomfortable branch is the one that says do that first. Really. A large share of fine-tuning projects are launched to fix problems that a better prompt, a few well-chosen examples and structured outputs would have solved in an afternoon — and the fine-tune then bakes the unexamined prompt's flaws into weights.

What LoRA actually does

If you do fine-tune, you will almost certainly use LoRALow-Rank Adaptation: instead of updating a weight matrix, you learn a small low-rank correction to add to it. The maths is worth understanding, because rank is the knob people set by superstition.

A transformer weight matrix W has shape (d, d). For d = 4096 that is 16.7 million parameters — per matrix, and there are many. Full fine-tuning updates all of them plus optimiser state, which is where the memory goes.

LoRA freezes W and learns two thin matrices instead:

At step zero the correction is exactly zero — B is initialised to zeros — so training starts from the base model's behaviour rather than from noise. Formally, with A of shape (r, d) and B of shape (d, r):

h = W x + (alpha / r) * B A x

Parameter count drops from d^2 to 2dr. At d = 4096, r = 8:

full:  4096 * 4096   = 16,777,216
LoRA:  2 * 4096 * 8  =     65,536     (0.4%)

import torch
import torch.nn as nn


class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=8, alpha=16, dropout=0.05):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False              # freeze the original weights

        self.A = nn.Parameter(torch.randn(r, base.in_features) * 0.01)
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        self.scale = alpha / r
        self.drop = nn.Dropout(dropout)

    def forward(self, x):
        return self.base(x) + self.drop(x) @ self.A.T @ self.B.T * self.scale

Choosing rank. The rank is the dimensionality of the change you are allowed to make.

Higher rank is not "better" — it is more capacity to overfit and more memory. Start at 8 or 16. alpha is conventionally 2r; the ratio alpha/r is the effective multiplier on the adaptation, so scaling both together keeps behaviour comparable.

Which layers. Attention projections only (q_proj, v_proj) is a good default for style. Adding k_proj, o_proj and the MLP projections gives more capacity when you are teaching genuinely new task structure, at roughly 3× the adapter size.

Cost, honestly

Rough orders of magnitude for a 7–8B open model, so you can sanity-check a business case:

Note the per-call row. RAG's ongoing cost is real: 4,000 tokens of retrieved context on every call adds up fast at scale. Prompt caching removes much of it for stable prefixes, but retrieved context is by definition not stable. This is the one strong economic argument for fine-tuning a small model on a high-volume narrow task — and it is a cost argument, not a quality one.

The dominant cost of fine-tuning is not GPU time. It is producing a few thousand examples that are correct, consistent and representative. Teams underestimate this by roughly a factor of five.

Doing both, which is usually the answer

For a mature product the two are complementary, not competing:

Fine-tuned small model  -->  knows your format, tone, refusal policy,
                             output schema, domain vocabulary
        +
RAG                     -->  supplies today's facts, with citations

The fine-tune handles how to answer; retrieval handles what is true right now. A support assistant fine-tuned on 3,000 approved transcripts writes like your best agent; RAG over the current knowledge base ensures it writes about the current product.

One non-obvious trap: fine-tune on examples that include retrieved context, not on bare question-answer pairs. Otherwise you train the model to answer from its weights and then, at inference, hand it context it was never taught to prioritise. The training example should look exactly like the production request.

# Right: the training example mirrors the production prompt shape
{
    "messages": [
        {"role": "user",
         "content": "<context>...retrieved passages...</context>\n\nQ: ..."},
        {"role": "assistant",
         "content": "...house-style answer citing the context..."},
    ]
}

The cheaper ladder, before you fine-tune anything

Work down this list. Stop at the first rung that solves the problem.

1. A better prompt. Specific, with the failure modes named. Most "the model can't do X" is "the model wasn't told what X means."
2. Few-shot examples. Three to five well-chosen examples beat three paragraphs of instructions for format tasks.
3. Structured outputs. If the problem is malformed JSON, schema enforcement solves it completely and costs nothing.
4. Prompt caching. If the problem is cost from a long stable prefix, cache it — roughly a 90% discount on the cached portion.
5. A more capable model at lower effort. Often beats a weaker model at high effort on both quality and cost.
6. RAG. If the problem is missing knowledge.
7. LoRA fine-tune. If the problem is behaviour at scale, or unit cost on a narrow task, and 1–6 genuinely did not close it.

import anthropic

client = anthropic.Anthropic()

# Rungs 1-4 in one request: specific prompt, cached stable prefix,
# schema-constrained output.
resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=4000,
    system=[{
        "type": "text",
        "text": HOUSE_STYLE_GUIDE,            # long, stable, cacheable
        "cache_control": {"type": "ephemeral"},
    }],
    output_config={
        "effort": "medium",
        "format": {"type": "json_schema", "schema": TICKET_SCHEMA},
    },
    messages=[{"role": "user", "content": user_ticket}],
)

print(resp.usage.cache_read_input_tokens)     # verify the cache is being hit

Teams that walk this ladder in order fine-tune far less often, and the fine-tunes they do run are aimed at problems fine-tuning actually solves.

Frequently asked questions

Can I fine-tune a model on my company documentation?

You can, and it will mostly disappoint you. The model learns your writing style rather than your facts, then fabricates confidently in that style. Use RAG for documentation. Fine-tune only if you also need a specific behaviour that prompting could not produce.

Does fine-tuning add new knowledge to a model?

Weakly and unreliably. Enough exposure shifts factual associations, but there is no guarantee of recall, no citation, no cheap update path, and a real risk of degrading unrelated abilities. Retrieval is the right mechanism for facts.

How many examples do I need for LoRA?

For style and format, 500–2,000 high-quality examples often suffice. For new task structure, expect several thousand. Consistency matters far more than volume — 500 clean examples beat 5,000 noisy ones, and inconsistent labels actively teach the model to be inconsistent.

Is RAG cheaper than fine-tuning?

Cheaper to build and to update; often more expensive per call, because retrieved context inflates every request. At low volume RAG wins economically. At very high volume on a narrow task, a fine-tuned small model can win on unit cost.

What is catastrophic forgetting?

When fine-tuning on a narrow dataset degrades unrelated abilities the model previously had. LoRA reduces it substantially — base weights are frozen, the adapter is low-rank — but does not eliminate it. Always evaluate on general capability tasks, not only on your target task.

Next reading on Sythra Articles

• Building RAG properly: chunking, evaluation and reranking
• LLM evaluation: grading a model with no right answer
• Production ML architecture