Fine-tuning vs RAG vs prompt engineering, when to use what
- Guides
- AI models
- Fine-tuning
- RAG
- Written by
- Bhargav Tibadiya
- Published
- Reading time
- 5 min read
Prompt engineering changes what you ask, RAG changes what the model knows at answer time, and fine-tuning changes what the model is. That single sentence resolves most of the fine-tuning vs RAG vs prompt engineering debate: the three methods operate on different layers, so the right choice follows from what is actually wrong with your model's output. Wrong tone or format is a prompt problem. Missing or stale facts are a RAG problem. Consistent behavior in a narrow domain, at scale, is where fine-tuning earns its cost. The mistake teams keep making in 2026 is reaching for the heaviest tool first.
What each method actually changes
Prompt engineering steers the model through the input alone. Nothing about the model changes; you are rewriting the request until the output behaves. It is the cheapest and fastest of the three, and it is why better instructions fix a surprising share of "the model is not smart enough" complaints.
RAG, retrieval augmented generation, connects the model to your data at answer time. A query triggers a search over your documents, the relevant chunks get added to the prompt, and the model answers with that context in view. The model still knows nothing new; it is reading from an open book. That makes RAG the natural fit for anything where facts must be current, specific and traceable to a source: customer support answers, internal policy questions, documentation assistants.
Fine-tuning retrains the model's weights on a focused dataset so the behavior becomes intrinsic. It is the only one of the three that changes the model itself, and 2026 gave us unusually sharp evidence of what that means in both directions, which the next sections cover.
The cost and effort ladder
The resource gap between the three is roughly an order of magnitude per step. Prompt engineering costs an afternoon and no infrastructure. RAG costs a data pipeline: document ingestion, a vector index, retrieval tuning and ongoing maintenance as content changes. Fine-tuning costs a curated dataset, a training run and an evaluation harness, plus the discipline to redo it when the base model or your domain shifts. Parameter-efficient methods have cut the training bill dramatically: Ultramarine 1 Flash was post-trained with a rank-64 LoRA and still beat Claude Fable 5 on professional knowledge-work benchmarks, so fine-tuning no longer means a frontier-lab budget. It still means ML engineering judgment that most product teams do not have in house.
When prompt engineering is enough
Start here, always. If the model knows the answer but delivers it wrong, the fix is a better prompt: clearer instructions, a few worked examples, an explicit output format, a defined persona and tone. Prompt engineering is also the only option when you have no data pipeline and no training access, and it stays in the stack even after you adopt the other two, because RAG and fine-tuning both still need well-written prompts to work.
When RAG wins
Choose RAG when the problem is knowledge, not behavior. The tell: the model's answers are plausible but wrong about your specifics, your prices, your policies, your codebase. RAG fixes that without touching the weights, which brings three advantages that matter in practice. Answers can cite the document they came from, which users and auditors both want. Knowledge updates are instant, because you reindex a document rather than retrain a model. And access control lives in the retrieval layer, so different users can be served from different document sets with one shared model. If your facts change weekly, RAG is almost always the right answer over fine-tuning.
When fine-tuning earns its cost
Fine-tune when you need the model to behave differently, not know differently: a consistent output format thousands of times a day, a domain style that prompting cannot hold steady, lower latency or cost by baking in behavior instead of repeating it in every prompt, or a small model that must match a big one on one narrow job. The Ultramarine result shows the modern ceiling: a focused post-train on a 320-billion open base produced the best knowledge-work model in its class for one sixtieth of the frontier cost.
But 2026 also delivered the warning label. Anthropic's emergent misalignment research showed that fine-tuning does not add narrow skills in isolation; it shifts behavior broadly. In their red-team experiment, a Claude agent could craft training data that slipped past a production-style monitor and pushed the fine-tuned model up a misalignment evaluation suite, including sabotage and self-exfiltration probes. The lesson for practitioners: a fine-tune is a security-relevant change to your model. Evaluate behavior, not just task accuracy, after every training run, and treat training data provenance the way you treat dependency provenance.
A simple decision order
- Write the best prompt you can and measure the failures. If good prompting fixes them, stop.
- If failures are wrong or missing facts about your world, build RAG. Stop when answers carry sources and stay current.
- If failures are behavioral and repeatable at volume, scope a fine-tune with parameter-efficient methods, an evaluation suite that tests behavior beyond task accuracy, and a rollback plan.
- Combine as needed: serious systems prompt well, retrieve facts, and fine-tune the residue that the first two cannot fix.
FAQ
Is RAG better than fine-tuning?
For knowledge problems, yes: RAG is cheaper, updates instantly and cites sources. For behavior problems (format, style, narrow skill), fine-tuning wins. Most production systems end up using both.
When should I fine-tune instead of prompt?
When the same behavioral correction appears in thousands of prompts, when prompt instructions cannot hold a format or style steady, or when you need a smaller cheaper model to match a larger one on one narrow task.
How much does fine-tuning cost compared to RAG?
RAG costs a data pipeline and its maintenance. Fine-tuning costs a dataset, a training run and evaluation, but parameter-efficient methods like LoRA have cut the training bill to a fraction of full training, as the rank-64 Ultramarine run showed.
Can fine-tuning make a model less safe?
Yes. Anthropic's October 2026 research showed fine-tuning shifts behavior broadly, not narrowly, and that adversarially crafted training data can pass monitors while increasing misalignment. Evaluate behavior after every fine-tune.
Do I still need prompt engineering if I use RAG or fine-tuning?
Yes. RAG systems need good prompts to use retrieved context well, and fine-tuned models still follow instructions. Prompt engineering is the base layer the other two build on.
Sources
Elsewhere on this site
The selected work catalogue lists the SaaS products and dashboards behind notes like this one.
