RAG or fine-tuning? Start with what the AI gets wrong

RAG retrieves information for a model to use when generating an answer. Fine-tuning adapts a model through additional training. Use retrieval when the task needs current or organisation-specific information; consider fine-tuning when a stable task requires different behaviour or output patterns. Some systems need both.

The choice starts with the failure you need to fix: what is wrong with the answers you are getting?

A system that cannot find the latest policy has a different problem from one that finds the policy but consistently writes in the wrong format. Buying a more elaborate architecture before separating those failures can add cost without improving the work.

For an enterprise architect choosing how to improve a policy assistant, this distinction already exists inside commercial platforms. Amazon Bedrock offers both model customisation and Knowledge Bases for retrieval. Its retrieval service can ingest source material and return supporting citations. Buying those capabilities does not decide which documents are authoritative or who may see them; those remain design decisions for the organisation.

That matters when a supplier proposes fine-tuning as the answer to stale information. Ask the team to demonstrate a failed answer, identify the information available at generation time and explain which intervention would change that failure. The architecture should follow the diagnosis.

Two interventions at different stages

Retrieval-augmented generation, usually shortened to RAG, brings retrieved information into the context used to generate an answer. Fine-tuning uses additional training to adapt a model. One changes the information supplied at response time; the other changes the model through training.

AWS’s comparison describes RAG as a starting point for questions over custom documents and notes that retrieval and fine-tuning can be combined. They are not mutually exclusive purchases.

For an enterprise buyer, the practical distinction is maintenance. When a travel policy changes, a retrieval system needs access to the current document and must stop serving obsolete material. A trained model should not be treated as a document repository that automatically acquires each policy revision.

Observed problem	First place to investigate	What would demonstrate progress?
The answer uses an obsolete policy.	Source currency, indexing and retrieval.	The answer reflects the approved current version.
The right evidence is available, but the output ignores the required structure.	Instructions, examples and possibly fine-tuning.	The output follows the structure on unseen tasks.
The answer draws on documents the requester should not see.	Permissions and the retrieval access boundary.	Restricted material cannot enter that requester’s context.
The answer remains unreliable despite relevant evidence.	Task design, model capability and evaluation.	Accuracy improves on representative difficult cases.

Retrieval makes the knowledge pipeline part of the product

Imagine an internal assistant answering questions about purchasing rules. Its value depends on more than finding a document containing the words “purchase order”. It needs the right business unit, jurisdiction, policy version and exception. A plausible answer assembled from several incompatible policies can be worse than an explicit request for clarification.

The evaluation therefore needs cases where documents disagree, where an answer is absent and where the requester lacks access. A successful search demonstration does not answer those questions. Neither does a citation by itself: the cited passage must support the answer.

This connects the architecture decision to data preparation and permissions. The retrieval layer inherits the organisation’s information problems unless someone resolves them.

Fine-tuning needs a stable target

Fine-tuning becomes a more plausible candidate when the team can describe a recurring behavioural problem and supply suitable examples of the desired result. That might involve a specialised response structure or a consistent task pattern. It should compete with simpler changes to instructions and examples, not bypass them.

A useful comparison gives each approach the same held-out tasks. If improved instructions meet the requirement, the additional training and maintenance may be unnecessary. If fine-tuning performs better, the team still needs to know whether the improvement is large enough to justify the extra work.

Keep training examples separate from the evaluation cases used to judge the result. Otherwise a system can appear successful because it has been optimised around the very examples presented as evidence of general performance.

Buy the improvement, not the architectural label

A combined approach can be sensible: retrieval supplies current material while a fine-tuned model handles a specialised task. It also creates more components to maintain and more places to investigate when an answer goes wrong.

Start with a small collection of genuine failure cases. Identify whether the missing ingredient is information, behaviour, permission enforcement or task capability. Then compare the simplest credible remedies. The right architecture is the one that fixes the observed problem at an acceptable operating cost, not the one with the most impressive acronym.