4  Running models in the cloud and on your own GPUs

When to leave the API, and what it costs

Every call so far has gone to a managed API. We send text to OpenRouter, a model somewhere answers, and we pay per token. For most of this book that is exactly right: it is the fastest way to build, it requires no hardware, and the per-call cost is negligible. It is not, however, the only place a model can run, and there are sound reasons to move. This chapter maps the three places a model can run, gives us a way to decide between them on cost, and shows the one technical fact that makes switching painless: a self-hosted open model can speak the same API as OpenRouter, so the code we have already written does not change.

NoteIn the running system

Every triage call and document answer our desk makes runs on a model somewhere, and in this chapter we decide whether that somewhere is a managed API or our own GPU (graphics processing unit). At real ticket volume the crossover here is the difference between a few cents a month and a few thousand.

NoteSetup for this chapter

Run in the gaba-core environment. The only code that executes here is a cost calculation, which needs no API key. We show the model-serving code without running it, as it requires a GPU this book’s build machine does not have. The comments tell you exactly what to do on a machine that does.

import pandas as pd
import matplotlib.pyplot as plt

4.1 Three places a model can run

A model can run in one of three places, and the choice is a trade-off between effort, control, and cost.

Managed API Rented cloud GPU Your own hardware
Example OpenRouter, OpenAI, Anthropic Colab (hosted notebooks), Vast.ai (GPU rental), a cloud VM (virtual machine) with a GPU A workstation or server you own
You manage Nothing The machine and the model server Everything, including the hardware
Time to first call Minutes An hour Days
Cost structure Per token Per hour the GPU is on Up-front, then electricity
Data leaves your control Yes, to the provider To the cloud provider No
Best when Building, low to moderate volume Bursty heavy jobs, fine-tuning, custom models Strict data rules, steady heavy volume

We have been in the first column the whole time. The rest of this chapter is about when and how to move right.

4.2 When to leave the API

There are five reasons to stop sending your work to someone else’s model, and only one of them concerns saving money.

Privacy and compliance. This is the most common reason in business settings. If your data cannot leave your premises, whether customer records under strict contracts, patient information, or anything a regulator treats as sensitive, then a managed API is unavailable regardless of price. Running the model yourself keeps the data on hardware you control. We return to the rules that force this in Appendix D.

Cost at scale. Per-token pricing is cheap until the volume is enormous, at which point a GPU you rent by the hour and keep busy can be cheaper than paying per token. We work out exactly where that crossover sits in the next section. In short, it is much further out than people expect for cheap models, and much closer for expensive ones.

Latency and control. A model on your own hardware has no network round trip to a provider and no queue you share with other customers. For a real-time product where every hundred milliseconds matters, that control can be worth the trouble.

Customization. A fine-tuned model (Part VI) is yours. Some providers will host fine-tunes for you, but if you have trained a model on your own data, serving it yourself is often the simplest path.

Working offline. Some environments have no internet at all. A local model is the only option.

Four of the five reasons have nothing to do with the per-token price. People reach for self-hosting expecting to save money and find that the real value was control. This distinction matters because the cost calculation, which we turn to now, often points back to the API.

4.3 The cost crossover

Here is the trade-off in numbers. A managed API charges per token, so its monthly cost grows with how much you use it. A rented GPU charges per hour whether or not you use it, so once you commit to keeping one running, its monthly cost is roughly flat. The question is at what monthly volume the rising API line crosses the flat GPU line.

# Illustrative assumptions, not quoted prices. Update them for your case.
GPU_HOURLY = 0.75          # a mid-range cloud GPU, USD per hour
HOURS_PER_MONTH = 730      # keeping it running around the clock
gpu_monthly = GPU_HOURLY * HOURS_PER_MONTH

# Three API price points, in USD per one million tokens, standing in for a
# cheap small model, a mid model, and a premium model.
api_prices_per_m = {"cheap model": 0.20, "mid model": 1.00, "premium model": 5.00}

# Utilization: the share of the provisioned GPU time that does useful work.
# Idle hours still appear on the bill, so at 50% utilization each useful
# token carries twice the flat cost, and the break-even volume doubles.
utilizations = [1.00, 0.50, 0.25]

rows = []
for name, price_per_m in api_prices_per_m.items():
    price_per_token = price_per_m / 1_000_000
    row = {"replacing": name, "API $/1M tokens": price_per_m,
           "self-host $/month": round(gpu_monthly)}
    for util in utilizations:
        breakeven_tokens = gpu_monthly / util / price_per_token
        row[f"break-even @{util:.0%}"] = f"{breakeven_tokens / 1e6:,.0f}M"
    rows.append(row)

pd.DataFrame(rows)
replacing API $/1M tokens self-host $/month break-even @100% break-even @50% break-even @25%
0 cheap model 0.2 548 2,737M 5,475M 10,950M
1 mid model 1.0 548 548M 1,095M 2,190M
2 premium model 5.0 548 109M 219M 438M

Read the break-even columns as “you need at least this much monthly volume before a flat-cost GPU beats paying the API per token.” Against a cheap model the break-even is enormous, in the billions of tokens a month, which is why for small models the API almost always wins on pure cost. Against a premium model the break-even arrives far sooner, because each token you stop buying is worth much more. The utilization columns turn the usual hand-wave about idle GPUs into arithmetic: a GPU doing useful work half the time still bills for every hour, so the break-even doubles, and at a quarter utilization it quadruples. Real workloads with peaks, troughs, and latency headroom rarely sit near the 100 percent column.

TipWith an AI coding tool

Adapting this break-even script to your own situation, with current GPU rates, your provider’s actual token prices, and more volume points, is mechanical work an assistant handles in one pass. The number to read in its output is the utilization you claimed, because utilization is the input people flatter: a GPU that serves a business day’s traffic and idles overnight runs near thirty percent, and an assistant asked to update the prices will happily leave a hundred-percent assumption in place unless you correct it.

Below the crossing point, the API is cheaper and you should stay on it. Above it, a busy self-hosted GPU is cheaper, provided you can keep it busy. This last condition is the catch: the flat cost assumes the GPU runs all month. A GPU that sits idle half the time has doubled its real per-token cost, pushing the crossover out exactly as the utilization columns in the table show. This is why the realistic version of “should we self-host to save money” is usually “only at high, steady volume, and usually only when replacing an expensive model.” One development blurs the flat-cost picture: serverless GPU platforms now bill per second of actual GPU time, which makes the self-host line behave more like a per-token line at low utilization and softens the idle-GPU penalty, at a higher effective hourly rate. For everything else, the reasons to self-host are the other four from the previous section.

The assumptions above are illustrative, and yours will differ. Drag the sliders to your own GPU rate, API price, and realistic utilization, and watch the crossover move. Utilization divides the useful work without shrinking the bill: the effective self-host cost per useful token is the flat monthly cost over the tokens actually produced, so dragging utilization down pushes the flat line up and the break-even out.

Figure 4.1: The cost crossover at your own numbers. The flat line is the GPU’s monthly cost spread over the tokens it actually produces at the chosen utilization; the rising line is the API. The dashed marker is the break-even volume, and lowering utilization pushes it out in direct proportion.

One more layer makes the procurement decision concrete. “Self-host” covers two distinct pricing structures: a dedicated GPU that bills for the whole month, and a serverless GPU platform that bills per second of actual use at a higher rate. Here are the three ways to buy the same workload, priced at three utilization levels with named illustrative rates.

# Three ways to procure the same capability. Named illustrative rates; swap in
# real quotes from your providers before deciding anything.
DEDICATED_HOURLY = 0.75    # committed cloud GPU: billed whether busy or idle
SERVERLESS_HOURLY = 2.40   # serverless GPU: per-second billing, busy time only
TOKENS_PER_SECOND = 800    # what the GPU sustains while busy
API_PRICE_PER_M = 1.00     # the mid model from the table above

rows = []
for util in [1.00, 0.50, 0.25]:
    busy_hours = HOURS_PER_MONTH * util
    tokens = busy_hours * 3600 * TOKENS_PER_SECOND
    rows.append({
        "utilization": f"{util:.0%}",
        "tokens/month": f"{tokens / 1e6:,.0f}M",
        "dedicated GPU": f"${DEDICATED_HOURLY * HOURS_PER_MONTH:,.0f}",
        "serverless GPU": f"${SERVERLESS_HOURLY * busy_hours:,.0f}",
        "managed API": f"${tokens / 1e6 * API_PRICE_PER_M:,.0f}",
    })

pd.DataFrame(rows)
utilization tokens/month dedicated GPU serverless GPU managed API
0 100% 2,102M $548 $1,752 $2,102
1 50% 1,051M $548 $876 $1,051
2 25% 526M $548 $438 $526

The dedicated column never moves, which is what “flat” means. At full utilization the dedicated GPU is the cheapest by a wide margin, which is the crossover argument in one row. As utilization falls, the dedicated price per useful token climbs while the serverless and API bills shrink with the work, and below roughly a third utilization the dedicated machine loses to serverless billing in this table. The rates are illustrative, but the pattern generalizes: dedicated wins only when busy, serverless softens the idle penalty at a premium hourly rate, and the API, which here costs more than serverless at every level, brings zero operations work, no model serving, and no capacity planning, which is why low and bursty volume tends to stay there anyway.

4.4 Running an open model yourself

Suppose one of those reasons applies and you do host a model. The switch does not require rewriting your code. Tools like vLLM (a high-throughput open-source inference server) serve an open-weights model behind an OpenAI-compatible endpoint. That means the same gaba.call_llm we have used all along works against your own server, with one change: the base URL. On a machine with a GPU, you would start a server like this:

# On a GPU machine (not run here). Installs vLLM and serves an open model
# behind an OpenAI-compatible API on port 8000.
%pip install -q vllm
# Then, in a terminal:
#   vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000

And then point the client at it. Because the endpoint speaks the OpenAI protocol, the whole switch is one call to gaba’s set_base_url helper:

# Talk to your own server, with OpenRouter out of the path. Same code everywhere else.
from gaba.llm import call_llm, set_base_url

# Repoint the shared client at the local server. Local servers do not check
# the API key, so the helper sends a placeholder.
set_base_url("http://localhost:8000/v1")

result = call_llm("Summarize this ticket in five words.", model="meta-llama/Llama-3.1-8B-Instruct")
print(result.text)

This one call is the entire switch: the triage system from Chapter 2, the structured outputs from Chapter 3, and everything we build later will run against a self-hosted model with this one change, because we were careful to route every call through one wrapper. This is the payoff of the shared gaba package: the decision about where a model runs is isolated to a single function.

To obtain a GPU in the first place without buying one, the two common routes are a free notebook environment like Google Colab for experiments, and an hourly GPU rental like Vast.ai or a cloud provider for heavier or longer jobs. We use a rented GPU in earnest in Part VI, when we fine-tune a model.

WarningDon’t outsource this

An assistant will readily generate a vLLM command or a cloud-GPU config for you, and you should let it. What it cannot do is decide whether you should self-host at all. This decision depends on your data rules, your real volume, and your true utilization, and getting it wrong is expensive in a way no config file fixes. Make the build-versus-buy decision yourself, from the crossover above.

4.5 What fits where: quantization at deployment time

Before any measurement, one decision determines which models you can serve at all: how many bits each weight gets. Chapter 16 covers quantization from the training side; here is the deployment side, which is mostly a sizing table. Weights at fp16 take two bytes per parameter, int8 one, 4-bit half of one, and the rule of thumb adds a few gigabytes of overhead for the runtime and the KV cache (the model’s running memory of the tokens so far) that grows with context length.

Table 4.1: Sizing by precision, weights only; leave headroom for the KV cache. The 4-bit formats (NF4, GPTQ, AWQ) trade a little accuracy for size, and Chapter 16 explains them.
precision bytes/param 8B model what one 24 GB card serves typical use
fp16 / bf16 2 ~16 GB up to ~10B quality-sensitive serving
int8 1 ~8 GB up to ~20B the balanced default
4-bit (NF4, GPTQ, AWQ) 0.5 ~4 GB ~30B and beyond fitting the biggest model that works

One wrinkle on the sizing table: most large models today are mixture-of-experts (MoE) designs, and their cards advertise two parameter counts. A name like qwen3.6-35b-a3b means 35 billion parameters total but only about 3 billion active per token: a router picks a few specialist subnetworks for each step, so the remaining experts stay idle for that token. The two counts govern different resources: memory scales with the total (all experts must be loaded), while speed and compute scale with the active count, which is how an MoE can answer like a large model and run like a small one. When you size hardware from the table above, use the total; when you estimate throughput, the active count is the better guide.

The format names matter only at the boundaries of tools. GGUF is the file format of the llama.cpp family, which is what runs models on laptops and phones, and what Ollama wraps for one-command local serving. GPTQ and AWQ are pre-quantized checkpoint formats that servers like vLLM load directly. FP8 is the newer datacenter option on recent GPUs, close to fp16 quality at half the memory. The practical reading of the table is economic: a quantization step down either fits a model twice the size on the same card, or doubles the batch headroom for the model you already serve, and this chapter’s cost story scales with that headroom.

The quality loss from quantization is not in the table, because it is workload-specific: narrow classification degrades little at 4-bit while long-chain reasoning degrades at higher precisions, and the published averages are averages over benchmarks you do not run. Chapter 16 measures the ladder on this book’s own task to show the method; the deployment rule is to run your Chapter 9 evaluation set against the quantized model before switching, the same gate as any other model change.

4.6 Measured: one model, one GPU, real numbers

All the arithmetic above leans on one number we kept calling an assumption, tokens per second, so we measured it. We served Llama-3.1-8B-Instruct with vLLM on a single RTX 3090 (a consumer 24 GB card) and pushed 48 generation requests of about 200 tokens each through it at three concurrency levels. These numbers came from one afternoon and one GPU; treat the pattern as the finding and your own measurement as the number.

Table 4.2: Measured serving economics for an 8B open model on one consumer GPU. *At an illustrative $0.35 per GPU-hour rental rate.
concurrent requests generation throughput p95 latency cost per 1M output tokens*
1 50 tok/s 4.0 s $1.93
4 190 tok/s 4.2 s $0.51
16 566 tok/s 7.5 s $0.17

The table is the utilization story in miniature, and it explains where self-hosting economics come from. One request at a time, the GPU idles between token steps and the effective price is close to API rates. At sixteen concurrent requests, continuous batching (the server interleaving many requests so the GPU never waits) fills the idle space: throughput grows elevenfold while the p95 latency a user feels less than doubles. Serving stacks exist to keep the silicon busy, which is also why the crossover table above is so sensitive to utilization: a self-hosted GPU is cheap per token only when something keeps the queue full.

4.7 Measuring the deployment decision

A deployment choice has no accuracy metric, but it still deserves measurement, and a common mistake is to make it from the numbers on a pricing page.

If you are seriously weighing self-hosting, measure three things on a trial run: throughput (tokens per second the server sustains under your real load), cost per thousand tokens (the GPU’s hourly cost divided by the tokens it actually produces per hour, which usually sits far below its theoretical peak), and p95 latency (the slow tail your users feel, which an average does not show). A GPU that looks cheap at peak throughput can be expensive at your real utilization, which is the most common way self-hosting cost estimates go wrong. A defensible decision rests on your own throughput-and-utilization numbers, plus an accurate account of which of the five reasons is driving the move.

TipCost: the lesson of this chapter

For the small, cheap models this book uses by default, the managed API is almost always the cheaper option until volume is very large. Self-hosting pays for itself mainly at high steady volume or when replacing an expensive model, and is justified more often by the non-cost reasons: privacy, control, and customization. On hardware you already own, the same arithmetic reads as throughput and utilization: the GPU is cheap per token only while something keeps its queue full. Decide on the real driver, then check the crossover above before assuming self-hosting saves money.

ImportantCompliance: self-hosting as a data-residency answer

The strongest reason to run a model yourself is that your data is not allowed to leave your control. If you operate under rules that forbid sending customer or regulated data to a third party, a self-hosted model on hardware you govern can be the only compliant option, whatever the cost math says. Appendix D covers when those rules apply.

4.8 Choosing your model layer

In Part I, we called models through OpenRouter, used Instructor (the validation-and-retry library from Chapter 3) to get typed output, and weighed where a model should run. These are choices, and a real system makes them deliberately.

Capability Open or self-host Hosted or managed Choose by
Reaching a model OpenAI SDK against any endpoint, LiteLLM proxy OpenRouter, Portkey one provider against many behind one key
Structured output Instructor, Outlines provider JSON or structured mode validation-with-retry against constrained generation
Where it runs vLLM, Ollama, TGI (Text Generation Inference), self-hosted hosted APIs, open-model inference (Together, Fireworks, Groq), Bedrock, Vertex, Azure; serverless GPU (Modal, RunPod) the Chapter 4 cost crossover

Important

  • Build against the OpenAI protocol so you can switch models without rewriting calls, since code written against a single provider’s SDK must be rewritten before it can leave that provider (Chapter 2).
  • Route by task: a cheap model for the easy work, a reasoning model only where it pays (Chapter 2).
  • Let the crossover decide self-host against API; volume and utilization move the line, and preference has no bearing on it (Chapter 4).

Common failure points

  • Treating structured output as guaranteed, when without validation and a retry one malformed reply breaks the pipeline (Chapter 3).
  • Self-hosting a GPU you cannot keep busy, which doubles the real cost per token (Chapter 4).
  • Wiring code to one provider’s quirks, which turns a model swap into a rewrite.

The current open-source and vendor options for these are in the tooling-landscape appendix, dated and fuller; model and provider choice is Appendix E.

4.9 Exercises

4.9.1 Conceptual questions

  1. A team runs a moderate ticket volume through a cheap model and considers self-hosting “to save money”. Given the crossover, the most likely outcome is:

    1. The API stays cheaper until their volume grows far beyond today’s
    2. Self-hosting wins immediately, because a GPU has no per-token charge
    3. The two options cost about the same at any moderate volume
    4. Self-hosting wins as long as they rent the GPU and avoid buying one

  2. Which of these is not a reason that points toward self-hosting?

    1. Customer data is contractually forbidden from leaving your systems
    2. The deployment must keep working with no internet connection at all
    3. You want the lowest per-call price at small, intermittent volume
    4. You fine-tuned a model on your own data and need somewhere to serve it

  3. Switching gaba.call_llm from OpenRouter to a self-hosted vLLM server requires changing mainly:

    1. The Pydantic schemas, to match the open model’s output format
    2. The base URL the client points at
    3. Every call site in every chapter’s notebooks
    4. The prompt templates, which open models read differently

  4. A cloud GPU looks cheap at peak throughput, but your workload keeps it busy only 30 percent of the time. Your true cost per token:

    1. Stays the same, since the hourly rate is fixed by the rental contract
    2. Falls, because a lightly loaded GPU serves each request faster
    3. Becomes irrelevant, because providers do not bill for idle hours
    4. Roughly triples, because the idle hours still cost money

  5. Which figure best supports a self-hosting proposal to a finance sponsor?

    1. The GPU’s advertised peak tokens per second from the vendor
    2. Cost per thousand tokens at your measured real utilization
    3. The hourly sticker price of the GPU rental on its own
    4. The open model’s score on a public capability benchmark

  6. The chapter lists five reasons to leave the API. What does it stress about them?

    1. Only one of the five concerns money; the rest are about control
    2. All five apply only once volume reaches billions of tokens a month
    3. Privacy is the weakest, since providers will sign data agreements
    4. No single reason suffices; at least two must apply together

  7. Why did routing every call through the one gaba wrapper pay off in this chapter?

    1. The wrapper caches replies, so a self-hosted server receives fewer calls
    2. The wrapper translates between the OpenAI and vLLM protocols on the fly
    3. The wrapper retries failures, which self-hosted servers produce more of
    4. The decision about where the model runs is isolated to one function

  8. Why does the break-even volume arrive sooner when replacing a premium model than a cheap one?

    1. Premium models need larger GPUs, which raises the flat monthly cost
    2. Premium models produce more output tokens per request on average
    3. Each avoided token is worth more, so the flat cost is covered sooner
    4. Providers discount cheap models at volume, moving their line down

4.9.2 Build lab

On a machine with a GPU (or a rented one), follow the serving steps above to run an open model behind vLLM, then change the base URL and re-run the Chapter 2 triage against your own server. One practical hurdle: Meta’s Llama weights are a gated repository on Hugging Face, so request access on the model page and set HF_TOKEN in your environment before vLLM can download them, or pick an ungated model. Confirm the results are categories from the same list. If you have no GPU, write down the exact two lines you would change and why nothing else does.

4.9.3 Evaluate lab

Take the crossover model above and plug in your own numbers: a real GPU hourly rate you can rent, the API price of the model you actually use, and a realistic estimate of your monthly token volume. Report a single conclusion with the number behind it: at your volume, which is cheaper, and how far are you from the crossover?

TipProject ideas

You can now call a model through an API, ask for typed output that you can load into a dataframe, reason about what each call costs, and route easy work to a cheap model and hard work to a stronger one. These two projects exercise that stack on data you choose.

  • A typed extractor for free-text records. Turn a stream of messy text into structured rows with a schema and a validation retry: pull the category, sentiment, dates, and amounts from support tickets, emails, or job postings. Send the easy cases to a cheap model and the hard ones to a stronger model, and track cost per record against accuracy on a labeled sample. Data to try: a Kaggle support-ticket set, a public email corpus, or job postings.
  • A router that cuts cost without losing quality. Send every request to a cheap model first, detect the low-confidence answers with a validation check or a self-rated score, and escalate only those to a stronger model. Measure the cost per thousand records and the quality against always using the strong model, then find the escalation threshold that holds quality for the least spend. Data to try: the labeled task from the first project, or any classification set with a known answer key.

These records become the raw material for the rest of the book, starting with Part II, where we read documents at scale.

NoteWhere we go next

This closes Part I. We can call models, get structured data back, and reason about where they run. Part II turns to the first substantial application: reading documents at scale. Chapter 5 begins where every document pipeline begins, by extracting clean text from messy PDFs.