RAG vs. fine-tuning: choosing an enterprise LLM architecture

Sending a 150,000-token handbook with every question costs about $3,000 a month on Claude Sonnet 5.5 at 10,000 questions. Retrieving only the right passages costs about $160 on the same model, and prompt caching brings the handbook approach down to about $550 on Sonnet 5.5, or $340 on Haiku 4.5. The third option, fine-tuning, got harder to reach in 2026: OpenAI is winding down its fine-tuning platform.

So the old question, retrieve or retrain, now has three answers, and each wins somewhere. Many production systems combine two.

The short answer: retrieve facts, tune behavior

Use retrieval-augmented generation (RAG) to give a model knowledge, and fine-tuning to change how it behaves. Long context sits in between: it works well when the knowledge is small and stable enough to send with every request.

The research backs the first half of that rule. When Ovadia and colleagues compared the two methods for injecting knowledge, RAG consistently outperformed unsupervised fine-tuning, both for facts the model had seen in training and for entirely new ones. Models struggle to learn new facts from fine-tuning alone. Fine-tuning earns its place elsewhere: format, tone, classification and cost.

When retrieval (RAG) wins

• Your knowledge changes weekly or daily.
• Answers must cite a source document.
• Access rules differ by user or team.

RAG searches your documents at request time and gives the model only the passages it needs. That has three practical effects. Updating knowledge means re-indexing a document, not retraining a model. Every answer can point back to the passage it came from, which is what auditors, lawyers and analysts ask for first. And because retrieval runs per request, it can filter by the user’s permissions before the model sees anything.

Our Relm build shows the pattern. Analysts upload rent rolls, offering memorandums, P&Ls and leases; Relm reads them with OCR, indexes them for vector search, and links every finding in its reports back to the record it came from. That traceability is the reason to choose retrieval: a commercial real estate analyst will not act on a number nobody can check.

When long context is enough

If the material is small and stable, skip the retrieval pipeline and put it in the prompt. Anthropic’s guidance is specific: a knowledge base under about 200,000 tokens, roughly 500 pages, can be included in the prompt whole, and prompt caching makes that much faster and cheaper on repeated requests.

Long context has limits that matter in enterprise work:

• Cost per query: you pay for every token on every request, so a large context sent thousands of times a day gets expensive, even with caching.
• Attention: the “Lost in the Middle” study by Liu and colleagues found that models use information at the start or end of a long input much better than information buried in the middle.
• Permissions: one shared prompt cannot easily respect different access rules for different users.
• Citations: pointing to the exact source passage is harder when the model read everything at once.

Long context is a good fit for a policy assistant over one handbook, or a contract review tool that reads one agreement at a time. It is a poor fit for a company-wide assistant over thousands of permissioned documents.

When fine-tuning wins

• You need a consistent format, tone or classification behavior.
• Latency or cost demands a smaller model.
• The task is narrow and examples are plentiful.

Fine-tuning trains a model on examples of the inputs and outputs you want. It is the right tool when prompting cannot reliably produce a strict JSON schema, a house writing style, or a classification label across thousands of edge cases. It can also move a task that works on a large model to a smaller, cheaper and faster one, which matters for voice agents and high-volume pipelines.

OpenAI’s own guidance puts fine-tuning last in the sequence: start with evaluations and prompt engineering, which may be all you need, and fine-tune when prompting alone does not deliver the behavior or performance required. The same guide now notes that OpenAI is winding down the fine-tuning platform, so for new teams that last step happens somewhere else. We follow the same order. Fine-tuning before you have an evaluation set means you cannot tell whether it helped.

What OpenAI’s fine-tuning wind-down changes

It removes the easiest self-serve path to a tuned GPT model, and it makes “prompt, retrieve, evaluate, tune last” the default for anyone starting now. OpenAI’s pricing page, checked October 18, 2026, says the company “is winding down the fine-tuning platform.” The platform is no longer open to new users. Existing users can create training jobs for the coming months, and fine-tuned models stay available for inference until their base models are deprecated.

Three practical consequences for an architecture decision:

• A plan that depends on fine-tuning a GPT model needs an account that already has fine-tuning access, and an exit date for that model.
• A tuned model now carries a retirement risk tied to its base model. Keep the training data and the evaluation set, because they are what let you rebuild the behavior on another model.
• Teams that need tuning will do it with another provider or an open-weight model they host. That adds hosting and evaluation work, which is one more reason to prove that prompting and retrieval fall short first.

None of this changes the research above. Retrieval was already the better way to add knowledge; the wind-down only raises the cost of reaching for tuning early.

A decision table for enterprise LLM architecture

Can you combine RAG and fine-tuning?

Yes. Retrieval supplies the facts at request time, and a tuned or carefully prompted model supplies the format and behavior. The two solve different problems, so they stack rather than compete.

“Retrieval for facts, instructions for behavior, and tuning only where evaluations show a gap.”

A typical enterprise assistant retrieves the relevant passages, sends them in a context window large enough to keep surrounding detail, and relies on careful instructions, or occasionally a tuned model, for format and tone. The architecture question is about which part of the problem each method should own.

Research has gone a step further and tuned models to use retrieval better. RAFT, from Zhang and colleagues (2024), trains a model on questions paired with retrieved documents, some of them deliberately irrelevant, and teaches it to ignore the documents that do not help and to quote the passage that does. The authors report that it consistently improved performance on PubMed, HotpotQA and Gorilla datasets in domain-specific RAG. The lesson for a buyer: if a tuned model is part of the design, tune it on retrieval-shaped examples from your own documents, not on the documents alone.

What each approach costs to run at 10,000 queries a month

At 10,000 queries a month, RAG costs about $80 to $160 in model tokens, long context over a 150,000-token reference set costs about $340 to $550 with caching and about $3,000 without, and a fine-tuned small model with retrieval costs about $61. The table shows the arithmetic, using list prices checked on October 18, 2026.

Assumptions: each RAG query sends 6,000 input tokens of question, instructions and retrieved passages and returns 400 output tokens, the same figures as our RAG development cost model. Each long-context query sends the full 150,000-token reference set plus a 100-token question and returns 400 tokens. Traffic has gaps, so we assume the five-minute cache expires and is rewritten 1,000 times a month. Anthropic bills a five-minute cache write at 1.25 times the input price and a cache hit at 0.1 times, except on Claude Sonnet 5.5 and Opus 5.5, where a hit costs 0.05 times.

Two things the table hides. RAG and the tuned model also need a vector store and the engineering to build retrieval, while long context needs almost none; that build cost is what the per-query saving pays back. And a fine-tuned model still needs retrieval for facts, which is why it appears with RAG rather than alone. For how these run costs sit next to the build price of a whole agent, see AI agent development cost.

Latency follows the same pattern. Retrieval adds a search step, usually small next to generation. Prompts of hundreds of thousands of tokens slow the first token. A smaller tuned model is often the fastest option of all, which is why fine-tuning matters most in latency-sensitive products such as voice agents.

Retrieval quality decides how accurate a RAG system is

When a RAG system gives a wrong answer, the cause is usually that it retrieved the wrong passage, not that the model reasoned badly. Retrieval is worth engineering properly. Anthropic reported that adding context to each chunk before indexing, combined with keyword search, reduced top-20 retrieval failures by 49% in its tests, and by 67% with a reranking step. The exact numbers will differ on your data, but the lesson holds: chunking, hybrid search and reranking move accuracy more than swapping the model.

Build evaluations first

Whichever you choose, build evaluations first. A set of real questions with expected answers is what lets you swap models safely when a better or cheaper one ships.

It is the cheapest part of the project and the one that makes every later decision measurable.

If you are choosing an architecture for private data, our enterprise LLM and RAG development work starts with exactly this: your documents, your access rules and an evaluation set, before any model is chosen, and RAG development cost explains what drives the price. For why so many AI projects stall after the demo, read why AI pilots stall before production.