15  When to fine-tune (and when not to)

The customization ladder and choosing a model

Until now we have used models exactly as the provider released them, and changed their behavior only through the prompt and the context we supplied. Fine-tuning, changing the model’s own weights on examples of your task, is the next mechanism, and it is the one people reach for too early. In this chapter we make the decision itself: when fine-tuning is the right tool, when something cheaper will do, and how to choose which model to fine-tune or call in the first place. The technique comes in Chapters 16 and 17, and the purpose of this chapter is to ensure you turn to them only when you should. Still in the specialize stage of the Chapter 1 lifecycle, this part customizes the model itself.

NoteSetup for this chapter

Run in the gaba-core environment with an OPENROUTER_API_KEY. The code here is a small few-shot demonstration and a cost calculation; no GPU or training is involved. The how-to of training arrives in the next chapter.

from dotenv import load_dotenv
load_dotenv()

import pandas as pd

15.1 The customization ladder

There is an order to customizing a model’s behavior, from cheapest and fastest to most expensive and slowest. You should climb this ladder from the bottom, stopping at the first rung that works.

  1. Prompting. Write clearer instructions; this is free, instant, and often enough.
  2. Few-shot examples. Show the model a handful of input-output pairs in the prompt. It learns the pattern in context, with no training.
  3. Retrieval (RAG, retrieval-augmented generation). When the gap is missing knowledge, give the model the facts at run time. This is how you add information to a model’s answers, as discussed in Part II.
  4. Fine-tuning. When the gap is a consistent behavior, a format, a style, or a narrow judgment the model keeps getting slightly wrong, train it on examples. This is the last rung, because it is the most expensive to build and maintain.

Most problems are solved on rungs one through three. Fine-tuning is for the residue that prompting and retrieval cannot fix, and reaching for it first is a common over-engineering mistake in the field.

flowchart BT
    r1["1 Prompting<br/><i>free, instant</i>"] -->|"still not enough?"| r2["2 Few-shot examples<br/><i>one longer prompt</i>"]
    r2 -->|"gap is missing knowledge?"| r3["3 Retrieval, RAG<br/><i>facts at run time</i>"]
    r3 -->|"gap is consistent behavior?"| r4["4 Fine-tuning<br/><i>training, the last resort</i>"]
Figure 15.1: The customization ladder. Climb from the bottom and stop at the first rung that works; each step up costs more to build and more to maintain.

15.2 Few-shot usually comes first

Before fine-tuning a model to produce a custom format, try simply showing it the format. As an example, here we teach an invented ticket-tagging scheme (with a team code and a priority) using three examples and no explicit rules.

from gaba.llm import call_llm

examples = """Ticket: I was double charged.
Output: [BILLING|P2] duplicate charge dispute
Ticket: App crashes on upload.
Output: [TECH|P1] upload crash
Ticket: Where is my package?
Output: [LOGISTICS|P3] shipment status inquiry"""

new = "Ticket: Someone accessed my account without permission.\nOutput:"
result = call_llm(examples + "\n" + new, system="Follow the format shown in the examples exactly.")
print(result.text.strip())
[SECURITY|P1] unauthorized access

From three examples, the model produced a correctly formatted tag, and even chose a sensible new team code the examples never showed it. This is in-context learning, and it cost one call and no training at all. A great many tasks that appear to require a fine-tuned model are in fact one few-shot prompt away. The rule is to exhaust this rung before climbing higher, because it is hard to justify the cost of fine-tuning for a behavior the model will copy from three examples.

15.3 Evaluation: measure the rung before you climb

One success on one ticket is an anecdote. The decision this chapter is about needs a number: how reliably does the cheap rung produce the behavior, because that number is the baseline any fine-tune would have to beat. Format compliance is ideal for this, since a regular expression can score it without a judge.

Metric: format validity, the share of replies matching the [TEAM|Pn] tag pattern.
Test set: twelve tickets from the running corpus, none of them in the examples.
Baseline: rung one, a prompt that describes the format in words but shows no examples.

import re
from concurrent.futures import ThreadPoolExecutor
from gaba.data import load_tickets

TAG = re.compile(r"^\[[A-Z]+\|P[1-3]\]\s+\S+")
test_tickets = load_tickets().head(12)["text"].tolist()

def few_shot(ticket: str) -> str:
    return call_llm(examples + f"\nTicket: {ticket}\nOutput:",
                    system="Follow the format shown in the examples exactly.").text.strip()

def instructions_only(ticket: str) -> str:
    return call_llm(
        f"Ticket: {ticket}\nTag this ticket: a team code in capital letters and a "
        "priority from P1 to P3, both inside square brackets separated by a pipe, "
        "then a short summary.",
        system="You tag support tickets.").text.strip()

with ThreadPoolExecutor(max_workers=8) as pool:
    few_out = list(pool.map(few_shot, test_tickets))
    zero_out = list(pool.map(instructions_only, test_tickets))

n = len(test_tickets)
print(f"format validity, few-shot examples:  {sum(bool(TAG.match(o)) for o in few_out)}/{n}")
print(f"format validity, instructions only:  {sum(bool(TAG.match(o)) for o in zero_out)}/{n}")
print("\none instructions-only reply, for flavor:", zero_out[0][:100])
format validity, few-shot examples:  7/12
format validity, instructions only:  10/12

one instructions-only reply, for flavor: [FINANCE|P1] Duplicate charge

The pattern these two lines typically show is the ladder’s whole argument in miniature: describing a format in words leaves the model room to improvise, while three examples pin it down, and the sample reply shows what improvisation looks like, a preamble here, a missing bracket there. Two limits apply to the measurement. Format validity is not routing accuracy: a tag can be perfectly formatted and still send the ticket to the wrong team, and checking that would need labeled routes. And twelve tickets is only a screening test; it is enough to tell you whether the cheap rung is in contention, which is all this decision needs. If the few-shot line is at or near perfect, as it usually is for a format this simple, the case for fine-tuning the format away has to rest on something else, like cost at volume, which is exactly what the break-even arithmetic later in this chapter prices.

15.4 What fine-tuning is for, and what it is not

Fine-tuning changes the model’s weights, and thus what it does by default (i.e., without any examples in the prompt). This makes it the right tool for a specific set of needs and the wrong tool for a tempting one.

It is for: a consistent output format or house style you do not want to re-teach in every prompt; a narrow, high-volume task where a small fine-tuned model can match a large model’s quality at a fraction of the cost; a domain judgment (is this contract clause unusual, is this transaction suspicious) that is hard to specify in words but easy to show in thousands of labeled examples.

The common and expensive mistake is to use it for adding knowledge. Fine-tuning a model on your documents teaches it the style of them without reliably teaching it the facts in them, and it will still fabricate. When the need is “the model should know our data,” the answer is retrieval: knowledge goes in the context; behavior goes in the weights.

One vocabulary note before we move on: everything we call fine-tuning in this part is supervised fine-tuning (SFT), i.e., training on input-output examples. This includes training on formats, styles, and narrow judgments. Above this sit two further rungs for behavior that is easier to rank than to write down: preference tuning, most often DPO (direct preference optimization), which trains on pairs of answers where one is preferred over the other; and reinforcement fine-tuning, which replaces fixed labels with a reward or a grader as the training signal. Both are beyond the scope of this book, but the names matter because provider offerings increasingly include them.

The decision usually starts from a symptom, so here is the ladder indexed the way problems actually arrive.

Table 15.1: From symptom to rung. The decision starts at the symptom, and the rung follows from it; a common mistake is to choose the technique in advance.
the symptom you observe the rung to reach for
answers are wrong about your company’s facts retrieval (rung 3); knowledge goes in the context
output format or style drifts on some inputs few-shot examples (rung 2), measured as above
quality is inconsistent and the instructions are vague better prompting (rung 1)
the model fabricates document contents it was “trained on” retrieval (rung 3); fine-tuning will not fix this
a format, style, or narrow judgment survives rungs 1-3 and runs at high volume fine-tuning (rung 4), if the break-even clears
the behavior is easier to rank than to specify or label preference tuning, beyond this book

15.5 When fine-tuning pays

The strongest case for fine-tuning is economic: a small model you fine-tune can do one narrow task as well as a large model you call by API, and then every inference is cheaper. Training is a fixed up-front cost; the savings accrue per call. So fine-tuning pays once you run enough volume to repay the training.

# Illustrative figures. A large API model versus a small fine-tuned model
# you host, for one narrow high-volume task.
training_cost = 50.0          # one-time fine-tuning cost, USD
api_cost_per_call = 0.0008    # large model via API, per call
tuned_cost_per_call = 0.0001  # small fine-tuned model you host, per call

savings_per_call = api_cost_per_call - tuned_cost_per_call
breakeven_calls = training_cost / savings_per_call

print(f"savings per call: ${savings_per_call:.4f}")
print(f"break-even at: {breakeven_calls:,.0f} calls")
for monthly in [100_000, 1_000_000, 10_000_000]:
    api = monthly * api_cost_per_call
    tuned = monthly * tuned_cost_per_call  # training already amortized after break-even
    print(f"  at {monthly:>11,}/mo: API ${api:,.0f} vs fine-tuned ${tuned:,.0f}")
savings per call: $0.0007
break-even at: 71,429 calls
  at     100,000/mo: API $80 vs fine-tuned $10
  at   1,000,000/mo: API $800 vs fine-tuned $100
  at  10,000,000/mo: API $8,000 vs fine-tuned $1,000
TipWith an AI coding tool

The break-even arithmetic above, a savings-per-call subtraction, a division into the training cost, a loop over monthly volumes, is ordinary boilerplate that an assistant will draft correctly in one pass. The judgment it cannot supply is what to put into training_cost, api_cost_per_call, and tuned_cost_per_call: whether those are honest estimates of your own workload or numbers borrowed from someone else’s example. Read every input before you trust the crossing point the code draws for you.

Below the break-even volume, the fine-tune never repays its training and you should stay on the API. Far above it, the cheaper per-call cost dominates and fine-tuning wins clearly. The structure is the same as the self-hosting crossover in Chapter 4, and the lesson rhymes with it: customization is an investment that pays off only at scale, so the deciding question is “do we have the volume to make it worth the cost,” a number you can compute before you train anything, whereas “would fine-tuning help” cannot settle the decision on its own.

Two cautions apply to the inputs. The per-call figure for the model you host assumes serverless hosting or a well-utilized server; Chapter 4’s utilization catch applies in full, and an idle GPU you are paying for raises the true per-call cost fast. And the $50 is compute only: building the training set is usually the larger fixed cost, since labeling a few thousand examples typically costs more than the training run itself.

Compute it for your own case here: set the training cost, the two per-call prices, and (if you would forgo serverless hosting and run your own server) its monthly fixed cost, then read off the break-even. The hosting slider is Chapter 4’s utilization catch made adjustable: a server you pay for monthly reduces the per-call savings before they can repay anything.

The same arithmetic appears below as a picture: two cumulative-cost lines, one starting at zero and climbing steeply, one starting at the training cost and climbing gently, and the volume where they cross.

import numpy as np
import matplotlib.pyplot as plt

calls = np.linspace(0, 200_000, 201)
fig, ax = plt.subplots(figsize=(7, 4))
ax.plot(calls / 1000, calls * api_cost_per_call, color="#0969da",
        label="API model (starts at $0)")
ax.plot(calls / 1000, training_cost + calls * tuned_cost_per_call, color="#8250df",
        label=f"fine-tuned (starts at ${training_cost:.0f})")
ax.axvline(breakeven_calls / 1000, color="#bf8700", linestyle="--", linewidth=1)
ax.annotate(f"break-even:\n{breakeven_calls:,.0f} calls",
            xy=(breakeven_calls / 1000, breakeven_calls * api_cost_per_call),
            xytext=(breakeven_calls / 1000 + 12, 22), color="#57606a", fontsize=9)
ax.set_xlabel("calls (thousands)")
ax.set_ylabel("cumulative cost, USD")
ax.legend(frameon=False)
ax.spines[["top", "right"]].set_visible(False)
plt.tight_layout()
plt.show()
Line chart of cumulative cost in dollars versus number of calls in thousands. The API line starts at the origin with a steep slope; the fine-tuned line starts at fifty dollars with a shallow slope; a vertical dashed line marks where they cross.
Figure 15.2: We plot cumulative cost against call volume for the API model, which starts at zero, and the fine-tuned model, which starts at its one-time training cost and climbs more slowly; the dashed line marks the crossover volume past which the fine-tune is the cheaper system.

Left of the dashed line the purple line is higher, which means the training cost has not yet been earned back and the API is the cheaper choice. Right of it the lines have swapped for good, and every additional call widens the gap. The picture also makes the failure case easy to imagine: flatten the blue line toward the purple one and the crossing slides off the chart, which is the never-repays case the calculator above reports.

15.6 Choosing a model

Whether you fine-tune or just call an API, you have to pick a model. In 2026, the choice is a portfolio, because no single model wins on every task.

  • Closed frontier models (the strongest from the major labs) are best when you need the highest capability and can send your data to a provider. You cannot host them yourself or touch their weights directly, though most can be fine-tuned through the provider’s hosted service.
  • Open-weight models (Llama from Meta, Mistral from Mistral AI, Qwen from Alibaba, and others) are what you fine-tune and host. As of 2026 the capability gap to the frontier has narrowed substantially on many business tasks, though it varies by task and model, and they give you control, privacy, and the ability to change the weights.
  • The deciding factors, which are rarely the benchmark scores, are whether your data can leave your walls, whether the model’s license permits your commercial use (open-weight licenses differ sharply, and some restrict exactly the use you have in mind), and where the model and its data are allowed to run.

Hosted fine-tuning deserves its own mention, because for most teams reading this book it is the first fine-tuning path they actually take. As of 2026, OpenAI, Google, and Amazon Bedrock all offer hosted supervised fine-tuning of their models, and increasingly preference tuning as well: you upload examples, pay a modest training fee, and call the tuned model through the same API as before. The economics differ from self-hosting, since the recurring cost is a small per-token premium on the tuned model, with no server to keep busy and zero operations work, but the break-even logic above is unchanged; only the numbers you plug in move. Chapters 16 and 17 train open-weight models because that is where you see and control everything, and Chapter 16 also walks the hosted recipe step by step. The decision between the two routes comes down to a handful of differences:

Table 15.2: Hosted versus self-managed fine-tuning. The break-even arithmetic above prices the recurring premium; the rest is governance.
Hosted fine-tune Self-managed (Ch 16)
Who runs the GPUs the provider you
Training data leaves your boundary stays on your hardware
The artifact a model name, locked to the provider an adapter file you own
Pricing small fee + per-token premium forever compute once, then your serving costs
Deprecation risk the base model retires on their schedule yours to manage
Time to first result an afternoon, no infrastructure an afternoon, if the GPU already exists

The practical pattern is to route: a cheap model for the routine bulk, a strong model for the difficult cases, an open-weight model you host for the regulated data. Appendix E lays out a fuller selection rubric. The mistake to avoid is choosing one model for everything; the right answer is usually a small portfolio, each model assigned to the work it fits.

15.7 Making the decision

Fine-tuning is a decision before it is a technique, and the test of a good one is whether you can defend it. For any task you are tempted to fine-tune, you should be able to say: which rung of the ladder you are on and why the ones below it were not enough, whether the need is knowledge (use retrieval) or behavior (consider fine-tuning), and at your volume, whether the economics clear the break-even. If you can answer those three, you are ready for Chapter 16. If you cannot, the right move is to climb back down the ladder and try the cheaper rung you skipped.

WarningDon’t outsource this

An assistant will readily walk you through fine-tuning whenever you ask, precisely because you asked. It will not tell you that a three-line few-shot prompt would have sufficed, or that your real need is retrieval, in which case training would be wasted. The decision of whether to fine-tune at all is yours, and making it well is what saves the time and money in this entire part.

TipCost: the decision is the cheap part

This chapter’s only API spend is a few-shot call, a fraction of a cent. This is the point: the customization ladder’s first three rungs cost pennies to try, and the break-even arithmetic costs nothing at all. Running the numbers before training anything is cheap mistake-avoidance; the expensive path is training first and discovering the volume was never there. The same check protects owned hardware, where no invoice records the mistake and its cost appears as an idle GPU and unused capacity.

15.8 Exercises

15.8.1 Conceptual questions

  1. You want the model to answer questions about your internal documents. The right rung of the ladder is:

    1. fine-tuning on the documents, so their contents become part of the weights
    2. few-shot examples drawn from the documents in every prompt
    3. retrieval, because the gap is missing knowledge, which belongs in the context
    4. prompting, with the documents summarized into the system prompt

  2. Fine-tuning changes the model’s:

    1. context window, letting it hold more of your data at once
    2. weights, and with them what the model does by default
    3. tokenizer, so your domain’s terms become single tokens
    4. API endpoint, routing your calls to dedicated hardware

  3. A common and expensive mistake with fine-tuning is:

    1. training with too few labeled examples to move the weights meaningfully
    2. continuing to train after the loss has already stopped falling
    3. tuning the largest available model when a small one would have done
    4. fine-tuning to add knowledge; it teaches style and still fabricates

  4. Fine-tuning a small model to replace a large API model pays off when:

    1. volume is high enough for the cheaper inference to repay the training cost
    2. the task is broad enough to exercise the complete range of the model’s abilities
    3. the API model’s accuracy on the task falls below an acceptable threshold
    4. the provider raises its prices, regardless of your monthly call volume

  5. In 2026, the deciding factor between an open-weight and a closed model is usually:

    1. the benchmark scores, which the closed frontier models still win
    2. the size of the context window that each model family offers
    3. the parameter count relative to the GPU memory you own
    4. data residency, license terms, and where the model may run

  6. In the few-shot demo, the model tagged a security ticket correctly after seeing three examples of other categories. What does that show?

    1. the model must have been fine-tuned on this tagging scheme beforehand
    2. in-context learning: it copied the pattern and generalized it, with no training
    3. the three examples permanently changed the model’s default behavior
    4. the system prompt secretly contained the full rules of the scheme

  7. Before committing to fine-tune for a custom output format, the first measurement to run is:

    1. how often a few-shot prompt gets the format right on new inputs, the baseline a fine-tune must beat
    2. the training loss a fine-tune would converge to on your formatting examples
    3. the number of tokens the custom format adds to each model response
    4. the latency difference between the small model and the large one

  8. In the break-even arithmetic, what happens when the fine-tuned model is not cheaper per call than the API model?

    1. the break-even point simply moves further out but still exists
    2. volumes above a million calls a month still flip the comparison
    3. no volume repays the training; the answer is to stay on the API
    4. the training cost should instead be amortized over a longer period

15.8.2 Build lab

Take a task you currently solve with a fine-tuned model, or imagine one, and try few-shot prompting on it: write three to five examples and test on new inputs. Report whether few-shot was good enough, and if not, what specifically it failed at, which is the actual case for fine-tuning.

15.8.3 Evaluate lab

Plug your own numbers into the break-even calculation: a realistic training cost, the API per-call price of the model you would replace, and the per-call cost of a small model you would host. Find your break-even volume and compare it to your actual monthly volume. Report whether fine-tuning would pay for your workload, with the number behind the decision.

NoteWhere we go next

If the decision is to fine-tune, Chapter 16 shows how to do it affordably. Full fine-tuning of a large model is out of reach for most teams. However, LoRA (low-rank adaptation) and its quantized variant QLoRA make it possible to fine-tune a capable open model on a single GPU. We do that next, with real training and a real before-and-after measurement.