Fine-Tuning or Prompting? A Decision Guide for Small Teams

When a language model does not quite do what you need, the instinct is often to fine-tune it. Sometimes that is right. More often, a better prompt or better context solves the problem faster and cheaper. This guide sets out how to decide, with the limited time and data a small team usually has.

Three ways to adapt a model

• Prompting: instructions, examples and output format in the request. Instant to change, no training.
• Retrieval (RAG): fetch relevant documents at request time and include them in the prompt. Gives the model knowledge it did not have.
• Fine-tuning: train the model further on your own examples, changing its weights. Changes how it behaves by default.

The key distinction is this: retrieval adds knowledge, fine-tuning shapes behaviour. Fine-tuning is a poor way to teach a model facts that change, because you would have to retrain whenever they do.

Start with prompting, seriously

Prompting is often abandoned too early. Before considering anything else, try:

1. Clear instructions that describe the task, the audience and what to avoid.
2. A handful of good examples (few-shot), including tricky cases.
3. A defined output format, ideally structured output against a JSON schema.
4. Breaking a complex task into smaller steps, each with its own prompt.
5. A stronger model, to check whether the task is achievable at all.

Measure each change against an evaluation set of real examples. Without that set, you cannot tell whether any approach, including fine-tuning, actually helped.

Add retrieval when knowledge is the gap

If the model fails because it does not know your products, policies or documents, RAG is the fix. It keeps knowledge current, lets you cite sources and respects access control, none of which fine-tuning offers.

When fine-tuning earns its place

Fine-tuning is worth considering when:

• You need a consistent format, tone or style that prompting delivers only most of the time.
• Your prompts have grown very long with examples, adding cost and latency on every request.
• You want a small, cheap model to match a larger model on one narrow task, such as classification or extraction.
• You have a self-hosted model and need specialised behaviour that open weights can learn.

You also need data. A few hundred to a few thousand high quality input and output pairs is a common range for parameter efficient methods, but quality matters far more than quantity. Inconsistent labels teach inconsistent behaviour.

What fine-tuning costs

	Prompting	RAG	Fine-tuning
Time to first result	Hours	Days	Days to weeks
Data needed	A few examples	Your documents	Hundreds or more labelled pairs
Infrastructure	None extra	Search index, ingestion pipeline	Training jobs, model registry, serving
Changing it	Edit the prompt	Update documents	Retrain and re-evaluate
Typical risk	Inconsistency	Poor retrieval	Overfitting, lost general ability

With open models, LoRA and QLoRA make fine-tuning affordable on a single GPU for many model sizes, using libraries such as Hugging Face PEFT and TRL, Axolotl or Unsloth. Several hosted providers also offer fine-tuning for some of their models through their APIs. Either way, you take on a training pipeline, versioned datasets and a model that must be re-evaluated and possibly retrained when the base model is updated.

A simple decision path

1. Build an evaluation set from real cases.
2. Improve the prompt until it stops improving.
3. If the gap is missing knowledge, add retrieval.
4. If the gap is behaviour, consistency or cost at volume, and you have clean data, try fine-tuning a small model.
5. Compare all options on the same evaluation set, including cost and latency, and pick the simplest one that passes.

Many teams find that steps 2 and 3 are enough. When they are not, the evaluation set and the examples gathered along the way become the training data for step 4, so none of the earlier work is wasted.

DoneDeploy helps teams run this comparison on their own data and builds the training and serving infrastructure when fine-tuning turns out to be the right call. If you are unsure which path fits, talk to us.