import numpy as np
import torch
from datasets import load_dataset, Dataset
from transformers import (AutoModelForSequenceClassification, AutoTokenizer,
Trainer, TrainingArguments)
from peft import LoraConfig, get_peft_model16 LoRA and QLoRA in practice
Fine-tuning a model on a single GPU
In Chapter 15 we decided whether to fine-tune; this chapter shows how to do it affordably. Full fine-tuning updates every weight in the model, which for anything but a tiny model needs more memory than a single GPU has and produces a full-size copy of the model for every task. LoRA (Low-Rank Adaptation) sidesteps both problems: it freezes the original weights (keeps them fixed during training) and trains a small set of new ones, a fraction of a percent of the total, and typically matches full fine-tuning’s accuracy on task-specific fine-tunes like the one in this chapter. QLoRA (Quantized LoRA) goes further, storing the frozen model in 4-bit so a much larger model fits in the same memory. Together they put fine-tuning a capable model within reach of one GPU and one afternoon. We do a real fine-tune here and measure it against full fine-tuning, on the same data, to see that the small adapter gives up almost nothing.
NoteSetup for this chapter
Run this in the gaba-finetune environment on a machine with a GPU (see Chapter 4 for renting one). We train real models here, so the first run downloads DistilBERT (a small pretrained text model), the IMDB sample, and the QLoRA demo model from Hugging Face (about 4 GB combined). The task is sentiment classification, which stands in for any text-classification problem you have (support-ticket urgency, customer-comment sentiment, document risk flags, etc.). The LoRA recipe is identical, whatever the labels mean.
16.1 Tools in this chapter
| Tool | Why we use it here | Alternatives | Trade-off |
|---|---|---|---|
| PEFT | Hugging Face’s library for parameter-efficient fine-tuning, including LoRA | Axolotl, Unsloth, Llama-Factory (wrappers over it) | direct control against more setup |
| bitsandbytes | loads the base model in 4-bit so a larger one fits in memory (QLoRA) | GPTQ, AWQ (quantized for serving) | memory saved against a little compute |
transformers Trainer |
runs the training loop over the model and data | a hand-written PyTorch loop | batteries included against less visibility |
Part VI closes, in Chapter 17, with the customization stack. The tooling-landscape appendix lists the current options.
16.2 The data and the task
We fine-tune a small pretrained encoder, DistilBERT, to classify text by sentiment. For the sake of reproducibility, we use a public sentiment dataset, but at work this would be the running example’s labeled tickets, with urgency in place of sentiment. The recipe would not change by a line.
MODEL = "distilbert-base-uncased"
raw = load_dataset("stanfordnlp/imdb")["train"].shuffle(seed=0).select(range(2000))
texts, labels = raw["text"], raw["label"]
n_classes = len(set(labels))
tok = AutoTokenizer.from_pretrained(MODEL)
idx = np.random.RandomState(0).permutation(len(texts))
split = int(0.8 * len(texts))
def make_split(indices):
enc = tok([texts[i] for i in indices], truncation=True,
padding="max_length", max_length=96)
enc["labels"] = [labels[i] for i in indices]
return Dataset.from_dict(enc)
train_ds, test_ds = make_split(idx[:split]), make_split(idx[split:])
print(f"{len(train_ds)} train, {len(test_ds)} test, {n_classes} classes")1600 train, 400 test, 2 classesOne speed shortcut to note: max_length=96 truncates every review hard, which lowers accuracy somewhat but keeps each training run short. Since it depresses both arms of the comparison equally, it does not tilt the result.
16.3 What LoRA does
A weight matrix in the model is large. LoRA leaves it frozen and learns a small low-rank update beside it: two skinny matrices whose product has the same shape as the original but only a tiny number of free parameters. At inference, this update is added to the frozen weight. Since only these skinny matrices train, the number of trainable parameters drops by a hundred times or more, which is what makes the memory and storage manageable.
flowchart LR
inp([input]) --> w["W: frozen original weights<br/>d x d parameters"]
inp --> a["A: project down to rank r<br/>r x d, trains"]
a --> b["B: project back up<br/>d x r, trains"]
w --> plus(("+"))
b --> plus
plus --> outp([output])
What the low rank costs is easiest to see by hand. A full fine-tuning parameter is independent: change it and exactly one entry of the update changes. A LoRA parameter, by contrast, is coupled to its neighbors: every entry of B affects a whole row of the product, and every entry of A a whole column, so the update’s entries move together whether we want them to or not.
The chapter’s results show what that constraint costs in practice: almost nothing, on this task, at a hundredth of the parameters. How much that saves depends on only two numbers: the width of the weight matrix and the rank we choose.
We wrap our training in a function so we can run it both ways, with LoRA and without.
def metrics(pred):
return {"acc": (np.argmax(pred.predictions, axis=1) == pred.label_ids).mean()}
def fine_tune(use_lora: bool):
# Start each arm from a clean slate so the peak-memory reading below
# measures this run, not whatever the previous one left on the GPU.
torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats()
model = AutoModelForSequenceClassification.from_pretrained(MODEL, num_labels=n_classes)
if use_lora:
# Train only small low-rank adapters on the attention projections.
# r=8 sets the adapter rank; lora_alpha scales its contribution (2x the rank is a common default).
model = get_peft_model(model, LoraConfig(
task_type="SEQ_CLS", r=8, lora_alpha=16, target_modules=["q_lin", "v_lin"]))
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
args = TrainingArguments(
# num_train_epochs is the number of full passes over the training data.
output_dir="/tmp/ft", per_device_train_batch_size=16, num_train_epochs=3,
learning_rate=1e-3 if use_lora else 2e-5,
report_to=[], logging_steps=50, disable_tqdm=True)
trainer = Trainer(model=model, args=args, train_dataset=train_ds,
eval_dataset=test_ds, compute_metrics=metrics)
trainer.train()
peak_gb = torch.cuda.max_memory_allocated() / 1e9
return trainer.evaluate()["eval_acc"], trainable, total, peak_gb, model
TipWith an AI coding tool
Once you have written fine_tune() yourself, handing it to an assistant for a second pass, say, to add a new precision argument, is a good use of the tool. Read the refactor against the original with particular care for the two lines that reset CUDA’s peak-memory counter before each call, since those lines are what make the memory comparison table honest. An assistant asked to tidy the function has no reason to know those lines matter, and one that drops them will return code that runs and quietly reports the wrong peak-memory figure.
16.4 LoRA matches full fine-tuning
Now we run the comparison that justifies the whole technique: we fine-tune the same model on the same data twice, once with LoRA and once updating every weight, and look at both the accuracy and how many parameters each trained.
lora_acc, lora_trainable, lora_total, lora_mem, lora_model = fine_tune(use_lora=True)
full_acc, full_trainable, full_total, full_mem, full_model = fine_tune(use_lora=False)
import pandas as pd
pd.DataFrame({
"method": ["LoRA", "full fine-tuning"],
"accuracy": [round(lora_acc, 3), round(full_acc, 3)],
"trainable params": [f"{lora_trainable:,}", f"{full_trainable:,}"],
# Each run is scored against its own total: the LoRA-wrapped model carries
# extra adapter parameters, so sharing one total would misreport the
# full fine-tune as training less than 100% of its model.
"% of model trained": [f"{100*lora_trainable/lora_total:.1f}%",
f"{100*full_trainable/full_total:.1f}%"],
"peak train memory (GB)": [f"{lora_mem:.2f}", f"{full_mem:.2f}"],
}){'loss': '0.5464', 'grad_norm': '2.165', 'learning_rate': '0.0008367', 'epoch': '0.5'}
{'loss': '0.451', 'grad_norm': '1.072', 'learning_rate': '0.00067', 'epoch': '1'}
{'loss': '0.3647', 'grad_norm': '1.398', 'learning_rate': '0.0005033', 'epoch': '1.5'}
{'loss': '0.3729', 'grad_norm': '1.309', 'learning_rate': '0.0003367', 'epoch': '2'}
{'loss': '0.2638', 'grad_norm': '0.5627', 'learning_rate': '0.00017', 'epoch': '2.5'}
{'loss': '0.2705', 'grad_norm': '2.882', 'learning_rate': '3.333e-06', 'epoch': '3'}
{'train_runtime': '7.952', 'train_samples_per_second': '603.7', 'train_steps_per_second': '37.73', 'train_loss': '0.3782', 'epoch': '3'}{'eval_loss': '0.4603', 'eval_acc': '0.82', 'eval_runtime': '0.359', 'eval_samples_per_second': '1114', 'eval_steps_per_second': '139.3', 'epoch': '3'}
{'loss': '0.6641', 'grad_norm': '2.331', 'learning_rate': '1.673e-05', 'epoch': '0.5'}
{'loss': '0.483', 'grad_norm': '6.441', 'learning_rate': '1.34e-05', 'epoch': '1'}
{'loss': '0.3612', 'grad_norm': '5.605', 'learning_rate': '1.007e-05', 'epoch': '1.5'}
{'loss': '0.3293', 'grad_norm': '5.694', 'learning_rate': '6.733e-06', 'epoch': '2'}
{'loss': '0.231', 'grad_norm': '0.8641', 'learning_rate': '3.4e-06', 'epoch': '2.5'}
{'loss': '0.2247', 'grad_norm': '10.55', 'learning_rate': '6.667e-08', 'epoch': '3'}
{'train_runtime': '11.42', 'train_samples_per_second': '420.2', 'train_steps_per_second': '26.26', 'train_loss': '0.3822', 'epoch': '3'}
{'eval_loss': '0.4486', 'eval_acc': '0.8175', 'eval_runtime': '0.3159', 'eval_samples_per_second': '1266', 'eval_steps_per_second': '158.3', 'epoch': '3'}| method | accuracy | trainable params | % of model trained | peak train memory (GB) | |
|---|---|---|---|---|---|
| 0 | LoRA | 0.820 | 739,586 | 1.1% | 0.66 |
| 1 | full fine-tuning | 0.818 | 66,955,010 | 100.0% | 1.62 |
The two methods reach essentially the same accuracy, although LoRA trained on the order of one percent of the parameters that full fine-tuning did. This is the entire value proposition: nearly identical quality for a tiny fraction of the trainable weights, which means a fraction of the memory during training and, for each task, a tiny adapter file of a few megabytes to store and distribute, whereas full fine-tuning produces a full-size model copy per task.
The peak-memory column shows the training saving directly, with one caveat about its size: what LoRA eliminates is the optimizer state (the running averages an optimizer like Adam keeps for every trainable weight) and the gradients (the per-weight error signals that training computes) for the frozen weights, and on a model as small as DistilBERT the activations from the batch account for a large share of the footprint in both arms, so the gap here is real but modest. Because the optimizer-state share grows with the model, the same comparison on a billion-parameter model is the difference between fitting on one GPU and not fitting at all, as the QLoRA section shows.
One accounting note on the table: about 592,000 of the LoRA run’s trainable parameters are the new classification head, which both arms must train from scratch. The LoRA adapters themselves are only about 147,000 parameters, roughly 0.2 percent of the model.
You can keep one base model and a folder of small adapters, one per task, and swap them in as needed. This swapping pattern is now a serving strategy in its own right: providers run one copy of the base model and attach a different customer’s adapter to each request. This is how per-customer fine-tunes are served without per-customer GPUs. (A refinement called DoRA, which splits the update into a magnitude and a direction, is worth knowing by name when you meet it in library options.)
The few-megabytes claim is checkable right now: saving the LoRA-wrapped model writes only what trained, the adapters and the classification head, omitting the frozen base.
from pathlib import Path
lora_model.save_pretrained("/tmp/adapter")
adapter_mb = sum(f.stat().st_size for f in Path("/tmp/adapter").rglob("*") if f.is_file()) / 1e6
print(f"LoRA adapter on disk: {adapter_mb:.1f} MB")
print("full fine-tuned model on disk: ~265 MB (a complete copy of DistilBERT)")
# Free both trained models so later cells measure their own memory, not ours.
del lora_model, full_model
torch.cuda.empty_cache()LoRA adapter on disk: 3.0 MB
full fine-tuned model on disk: ~265 MB (a complete copy of DistilBERT)Putting a new task into use means copying a file the size of a photo; this is the adapter-folder pattern described above, measured on disk.
Two caveats temper this comparison. The accuracy difference between the two methods is small enough to sit within run-to-run variance on a task this easy, so read the result as “no measurable difference here.” Reading it as proof that LoRA equals full fine-tuning on every task would overstate what a single easy task can show. And the two arms use different learning rates (the learning rate is the size of each step the optimizer takes when updating weights), a higher one for LoRA. This difference is standard practice, and it does not distort the comparison: small adapters train well at a learning rate that would destabilize a full fine-tune. The robust claim survives both caveats, which is that LoRA reaches full fine-tuning’s quality here while training a hundredth of the parameters; to make that claim about your own task, run this same comparison on your data.
16.5 QLoRA: fitting a bigger model in memory
LoRA shrinks the trainable weights, but the frozen base model still has to sit in memory, and for a large model that alone can exceed a single GPU. QLoRA’s contribution is to store the frozen base in 4-bit precision, a quarter of the usual 16-bit footprint. Because the frozen base is only read during training, the precision loss barely matters, and the memory saving is substantial. What 4-bit storage actually does to the weights is simple to see: it rounds every value to the nearest of a small set of levels.
Here is the same model loaded both ways.
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
BIG = "Qwen/Qwen2.5-1.5B-Instruct"
torch.cuda.reset_peak_memory_stats()
m16 = AutoModelForCausalLM.from_pretrained(BIG, torch_dtype=torch.float16, device_map="cuda")
vram_16bit = torch.cuda.max_memory_allocated() / 1e9
del m16; torch.cuda.empty_cache(); torch.cuda.reset_peak_memory_stats()
nf4 = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16)
m4 = AutoModelForCausalLM.from_pretrained(BIG, quantization_config=nf4, device_map="cuda")
vram_4bit = torch.cuda.max_memory_allocated() / 1e9
print(f"16-bit weights: {vram_16bit:.2f} GB")
print(f" 4-bit weights: {vram_4bit:.2f} GB")
print(f"reduction: {vram_16bit / vram_4bit:.1f}x")16-bit weights: 3.11 GB
4-bit weights: 1.26 GB
reduction: 2.5x16.5.1 The precision ladder, measured
Four bits is one rung on a ladder, and the ladder matters because every model you deploy sits on it somewhere. Each parameter is a number, and how many bytes the number gets determines the memory arithmetic downstream: a seven-billion-parameter model is 28 GB of weights at fp32, 14 GB at fp16, 7 GB at int8, and 3.5 GB at 4-bit. These quantized weights are stored small but computed big, since each layer’s weights are dequantized to 16-bit on the fly for the matrix multiply, like a compressed file that is only extracted at the moment of use, so the precision of the arithmetic never drops to 4 bits. QLoRA’s “double quantization” even compresses the per-block scaling factors themselves, recovering another fraction of a bit per parameter.
Folklore has it that each rung down loses a little accuracy, but folklore is not a measurement. To measure, we run the ladder on our own model and task: the same Qwen model loaded at three precisions, each scored zero-shot on a slice of the chapter’s held-out reviews, while also recording memory and speed.
import time
def zero_shot_sentiment(model, sample_idx):
"""Greedy yes/no sentiment from the instruct model, scored against gold."""
correct = 0
t0 = time.perf_counter()
for i in sample_idx:
prompt = ("Review: " + texts[i][:400] +
"\nIs this review positive or negative? Answer with one word.")
msgs = [{"role": "user", "content": prompt}]
enc = qtok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
ids = qtok(enc, return_tensors="pt", truncation=True, max_length=256).to("cuda")
with torch.no_grad():
out = model.generate(**ids, max_new_tokens=3, do_sample=False,
pad_token_id=qtok.pad_token_id or qtok.eos_token_id)
reply = qtok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True).lower()
pred = 1 if "positive" in reply else 0
correct += pred == labels[i]
return correct / len(sample_idx), time.perf_counter() - t0
from transformers import AutoTokenizer as _AT
qtok = _AT.from_pretrained(BIG)
sample = [int(i) for i in idx[split:][:100]]
ladder_rows = []
configs = [
("bf16", dict(torch_dtype=torch.bfloat16)),
("int8", dict(quantization_config=BitsAndBytesConfig(load_in_8bit=True))),
("nf4", dict(quantization_config=BitsAndBytesConfig(
load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16))),
]
for name, kw in configs:
torch.cuda.empty_cache(); torch.cuda.reset_peak_memory_stats()
mdl = AutoModelForCausalLM.from_pretrained(BIG, device_map="cuda", **kw)
mem = torch.cuda.max_memory_allocated() / 1e9
acc, secs = zero_shot_sentiment(mdl, sample)
ladder_rows.append({"precision": name, "weights+overhead (GB)": round(mem, 2),
"accuracy (n=100)": acc, "reviews/sec": round(100 / secs, 1)})
del mdl
torch.cuda.empty_cache()
pd.DataFrame(ladder_rows)| precision | weights+overhead (GB) | accuracy (n=100) | reviews/sec | |
|---|---|---|---|---|
| 0 | bf16 | 4.32 | 0.87 | 17.3 |
| 1 | int8 | 3.07 | 0.87 | 3.9 |
| 2 | nf4 | 2.47 | 0.88 | 11.1 |
Read the three columns together, because the ladder is a three-way trade among memory, speed, and accuracy. Memory falls roughly as the bits do. Speed is where the folklore breaks down: in our run the int8 model is by far the slowest, because the bitsandbytes int8 path spends extra compute separating outlier values on every multiply, and even nf4 trails bf16, since dequantizing on the fly is work. The “quantized means faster” claim is true of inference engines with fused quantized kernels (the GPTQ and AWQ checkpoints a server like vLLM loads), whereas in a training-oriented loader like this one bitsandbytes saves memory at some cost in speed. And accuracy on this narrow task should barely move at all, which is the slightly deflating norm: a classification task with a clear answer survives 4-bit easily, while long-form generation and chains of reasoning degrade earlier, and the only way to know where your task sits on that spectrum is exactly this table run on your own evaluation set. The illustrative accuracy curves that circulate online (fp16 at 99.5 percent, int4 at 93) are averages over benchmarks you do not run; your number is the one that matters.
One more intuition explains why degradation, when it comes, comes suddenly. Rounding error is injected at every layer, and a transformer applies dozens of layers in sequence, so the size of any one layer’s error matters less than how the errors compound on the way through.
Weights are only one of four things competing for the GPU during training; the others are gradients, optimizer states, and activations, and which of them dominates depends entirely on how many parameters actually train. The anatomy below shows the budget that constrains every fine-tuning decision.
The anatomy explains why each technique exists: full fine-tuning is dominated by its optimizer states, LoRA shrinks those to a rounding error but still carries full-precision frozen weights, and QLoRA shrinks the weights too. The 4-bit version uses a fraction of the memory, and the gap widens for larger models, where the weights dominate. From there, QLoRA is simply LoRA applied on top of the 4-bit base: prepare the quantized model for training, attach the same low-rank adapters, and train as before, which is what we do here with one pass through the same data.
from peft import prepare_model_for_kbit_training
del m4; torch.cuda.empty_cache(); torch.cuda.reset_peak_memory_stats()
# Load the 4-bit base with a classification head, then attach LoRA adapters.
qmodel = AutoModelForSequenceClassification.from_pretrained(
BIG, num_labels=n_classes, quantization_config=nf4, device_map="cuda")
qtok = AutoTokenizer.from_pretrained(BIG)
qmodel.config.pad_token_id = qtok.pad_token_id
qmodel = prepare_model_for_kbit_training(qmodel)
qmodel = get_peft_model(qmodel, LoraConfig(
task_type="SEQ_CLS", r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"]))
def qwen_split(indices): # same data, retokenized for this model
enc = qtok([texts[i] for i in indices], truncation=True,
padding="max_length", max_length=96)
enc["labels"] = [labels[i] for i in indices]
return Dataset.from_dict(enc)
qtrainer = Trainer(
model=qmodel,
args=TrainingArguments(output_dir="/tmp/qlora", per_device_train_batch_size=8,
num_train_epochs=1, learning_rate=2e-4, bf16=True,
report_to=[], logging_steps=50, disable_tqdm=True),
train_dataset=qwen_split(idx[:split]), eval_dataset=qwen_split(idx[split:]),
compute_metrics=metrics)
qtrainer.train()
qlora_acc = qtrainer.evaluate()["eval_acc"]
qlora_mem = torch.cuda.max_memory_allocated() / 1e9
qlora_trainable = sum(p.numel() for p in qmodel.parameters() if p.requires_grad)
print(f"QLoRA accuracy after one epoch: {qlora_acc:.3f}")
print(f"peak GPU memory during training: {qlora_mem:.2f} GB"){'loss': '1.069', 'grad_norm': '21.87', 'learning_rate': '0.000151', 'epoch': '0.25'}
{'loss': '0.8458', 'grad_norm': '31.84', 'learning_rate': '0.000101', 'epoch': '0.5'}
{'loss': '0.7003', 'grad_norm': '48.92', 'learning_rate': '5.1e-05', 'epoch': '0.75'}
{'loss': '0.5958', 'grad_norm': '67.27', 'learning_rate': '1e-06', 'epoch': '1'}
{'train_runtime': '47.97', 'train_samples_per_second': '33.35', 'train_steps_per_second': '4.169', 'train_loss': '0.8026', 'epoch': '1'}
{'eval_loss': '0.5593', 'eval_acc': '0.755', 'eval_runtime': '3.36', 'eval_samples_per_second': '119.1', 'eval_steps_per_second': '14.88', 'epoch': '1'}
QLoRA accuracy after one epoch: 0.755
peak GPU memory during training: 3.66 GBOne epoch on our small sample is enough to show the recipe end to end: the base model’s weights stay frozen in 4-bit, only the adapters and the classification head train in regular precision, and the peak memory during training stays within what a modest single GPU offers, even though this model is twenty times DistilBERT’s size. This is the combination that lets a single 24 GB GPU fine-tune models that would otherwise need several, and it is why “fine-tune a 7-billion-parameter model on one consumer GPU” went from impossible to routine.
The three training runs for the chapter are presented side by side in a single table below.
pd.DataFrame({
"run": ["LoRA (DistilBERT)", "full fine-tune (DistilBERT)", "QLoRA (Qwen 1.5B)"],
"accuracy": [round(lora_acc, 3), round(full_acc, 3), round(qlora_acc, 3)],
"trainable params": [f"{lora_trainable:,}", f"{full_trainable:,}", f"{qlora_trainable:,}"],
"peak train memory (GB)": [f"{lora_mem:.2f}", f"{full_mem:.2f}", f"{qlora_mem:.2f}"],
"base precision": ["fp32", "fp32", "nf4 (4-bit)"],
})| run | accuracy | trainable params | peak train memory (GB) | base precision | |
|---|---|---|---|---|---|
| 0 | LoRA (DistilBERT) | 0.820 | 739,586 | 0.66 | fp32 |
| 1 | full fine-tune (DistilBERT) | 0.818 | 66,955,010 | 1.62 | fp32 |
| 2 | QLoRA (Qwen 1.5B) | 0.755 | 1,092,608 | 3.66 | nf4 (4-bit) |
One reading note: since the QLoRA row trains a different and much larger model for a single epoch, its place in the table is to show that a model twenty times the size of DistilBERT trained in a comparable memory budget, which is the point of storing the base in 4-bit. The apples-to-apples comparison, the first two rows, is the one we evaluate in the next section.
16.6 The hosted route: the same fine-tune without owning the loop
Everything above assumed a GPU that you control. Most teams run their first fine-tune through a hosted service, where the provider runs the training loop for you. At many companies, this is the only sanctioned route, as no infrastructure gets provisioned and the data pipeline is one API. We walk through the recipe step by step for OpenAI’s fine-tuning API, the one most often named in job descriptions. Google’s Vertex AI and AWS Bedrock follow the same pattern with different nouns. We show the code without running it, as it needs a direct provider key (OpenRouter does not proxy fine-tuning jobs) and each job costs real money.
The data format is the whole trick: the same examples we trained on locally, rewritten as chat transcripts, one JSON object per line.
import json
train_idx = [int(i) for i in idx[:split]]
with open("sentiment_train.jsonl", "w") as f:
for i in train_idx:
f.write(json.dumps({"messages": [
{"role": "system", "content": "Classify the review's sentiment. Reply positive or negative."},
{"role": "user", "content": texts[i][:1000]},
{"role": "assistant", "content": "positive" if labels[i] == 1 else "negative"},
]}) + "\n")Then make four calls: upload the file, start the job, poll until it finishes, and use the model it returns. The provider picks the GPUs, runs what is almost certainly a LoRA-style adapter internally, and returns a model name.
from openai import OpenAI
client = OpenAI() # direct OpenAI key; not the book's OpenRouter client
up = client.files.create(file=open("sentiment_train.jsonl", "rb"), purpose="fine-tune")
job = client.fine_tuning.jobs.create(
training_file=up.id,
model="gpt-4.1-mini-2025-04-14",
hyperparameters={"n_epochs": 2}, # the same dial we set locally
)
# Poll (a job on 1,600 examples typically finishes within the hour):
job = client.fine_tuning.jobs.retrieve(job.id)
print(job.status) # validating_files -> running -> succeeded
# When it succeeds, the fine-tuned model is just another model name:
reply = client.chat.completions.create(
model=job.fine_tuned_model, # e.g. ft:gpt-4.1-mini-2025-04-14:org::abc123
messages=[{"role": "user", "content": "The plot dragged but the acting was superb."}],
)
print(reply.choices[0].message.content)What you give up and what you get, against the QLoRA run we just did, is a clean trade to put in a memo. The hosted job removes every operational step this chapter performed: no environment, no VRAM (the GPU’s onboard memory) arithmetic, no checkpoint management, and the result autoscales behind the provider’s API from minute one. In exchange, the training data leaves your boundary (the compliance question from Appendix D applies to fine-tuning data doubly, since it is by definition your most curated text), the artifact is locked to the provider and priced as a per-token premium on every future call, whereas the local recipe produces a few megabytes you own, and the base model can be deprecated on a schedule the provider alone controls. As of this writing the training itself costs a few dollars per million training tokens, which for our 1,600 short reviews is negligible; the recurring inference premium is where the real cost accumulates, and Chapter 15’s break-even arithmetic prices exactly that trade. The sensible default for a first project: prototype hosted, and bring the fine-tune in-house with this chapter’s recipe when volume, residency, or deprecation risk justifies the move.
16.7 Evaluation: did the fine-tune help, and at what cost in parameters?
The evaluation here is built into the comparison, which is the right way to judge a fine-tune: accuracy against a baseline, paired with the cost of getting it.
Metric: classification accuracy on the held-out test set, and trainable-parameter count as the cost.
Test set: the 20 percent we held out, never seen in training.
Baseline: full fine-tuning, the strongest and most expensive option.
LoRA reached the baseline’s accuracy while training around one percent of its parameters. This is the result that matters: when an inexpensive method matches an expensive one on the metric you care about, you deploy the inexpensive one. The same evaluation discipline from Chapter 9 applies, only now the “system” is a set of weights, and the thing you are measuring is whether the fine-tune was worth it against the simpler option, exactly the question Chapter 15 told you to ask before training at all.
TipCost: training is a fixed cost, adapters are cheap
A LoRA fine-tune like this runs in minutes on a single GPU, a few dollars of rented compute or free on hardware you own. The adapter it produces is a few megabytes, so storing and serving many task-specific fine-tunes costs almost nothing on top of the one base model. The hosted alternative inverts this structure: its one-time training fee is similarly small, but the fine-tune is then paid for again on every future call as a per-token inference premium, which is the arithmetic Chapter 15’s break-even calculation prices. This is the cost structure that made fine-tuning, once the province of teams with clusters, something an analyst can do in an afternoon, and it is what turns Chapter 15’s break-even calculation in fine-tuning’s favor sooner than full fine-tuning ever could.
16.8 Exercises
16.8.1 Conceptual questions
LoRA cuts the number of trainable parameters by freezing the original weights and:
- Quantizing every weight down to 4-bit precision so each update takes less memory
- Deleting most of the attention layers and retraining the smaller model that remains
- Training two skinny low-rank matrices whose product corrects the frozen weights
- Training only the final classification head and leaving every other layer untouched
In the interactive demo, the rank-1 LoRA update could not change just the two target cells because:
- Every entry of B moves a whole row of the update, and every entry of A a whole column
- The frozen original weights override any update that touches cells they already determine
- Rank-1 matrices can only hold values that share a single sign across the whole update
- The click increment of 0.4 was too coarse a step to confine the change to single cells
In the chapter’s comparison, LoRA’s held-out accuracy came within a point of full fine-tuning’s. Our reading of this result is:
- LoRA is reliably more accurate than full fine-tuning and should replace it in every project
- Full fine-tuning was undertrained, and a few more epochs would have reversed the ordering
- The task was too easy for the comparison to mean anything about either method
- There is no measurable difference here, since the gap sits within run-to-run variance
The LoRA arm trained at a learning rate of 1e-3 while the full fine-tune used 2e-5. We treat this as:
- A flaw in the experimental design that quietly tilts the accuracy comparison toward LoRA
- Standard practice: adapters train well at a rate that would destabilize a full fine-tune
- An arbitrary library default that should have been held equal across both arms
- A sign that the small adapters need far larger steps to compensate for their lower capacity
What does QLoRA add on top of LoRA?
- Storing the frozen base model in 4-bit so a larger model fits in the same memory
- A second pair of adapters that captures whatever the first pair’s low rank misses
- A compression step applied to the adapters after training to shrink the saved files
- A distillation pass that transfers the fine-tuned model’s skill into a smaller model
Why does storing the base model in 4-bit barely hurt quality during QLoRA training?
- Rounding the weights toward a coarse grid acts as a regularizer that improves generalization
- The adapters are stored at the same precision, so the rounding errors cancel out
- Modern GPUs compute natively in 4-bit, so no precision is lost in the arithmetic
- The frozen base is only read during training, so a little rounding error is tolerable
A practical benefit of LoRA’s few-megabyte adapter files is that you can:
- Skip the held-out evaluation, because adapters that small cannot overfit the data
- Keep one base model and swap small per-task adapters in and out as needed
- Serve the fine-tuned model from a CPU, since adapters remove the GPU requirement
- Retrain the base model’s lower layers later without invalidating the adapters
In the chapter’s evaluation, full fine-tuning serves as the baseline because:
- It is the cheapest alternative on offer, playing the role the naive forecast plays elsewhere
- It shows the pretrained model’s accuracy before any task-specific training happens
- It is the strongest, most expensive option, so matching it cheaply is what LoRA must prove
- It is the only method whose accuracy does not depend on the choice of learning rate
16.8.2 Build lab
Change the LoRA rank r from 8 to 2 and to 32, and re-run the fine-tune at each. Report how accuracy and trainable-parameter count change with rank, and pick the rank you would use, justifying the trade between capacity and cost.
16.8.3 Evaluate lab
Add a zero-shot baseline: before any fine-tuning, evaluate the pretrained model’s accuracy on the test set (its classification head is untrained, so expect near chance). Put it in the table alongside LoRA and full fine-tuning. Report the gap fine-tuning closed, and decide whether LoRA’s share of that gain justifies its tiny cost.
NoteWhere we go next
Fine-tuning requires labeled examples, and often you do not have enough of them. In Chapter 17, we use a large model to generate training data for a smaller one, distilling the large model’s skill into a cheap fine-tuned model, and we show the one rule that keeps synthetic data from quietly poisoning the result.