24 Observability and drift

Watching a system that is live

A system that passed every test on the day it went live can be quietly wrong a month later. The model provider silently updates a model, the kinds of questions users ask shift, a prompt change three weeks ago degraded answers and no one noticed. Because production large language model (LLM) systems rarely break loudly, failure typically takes the form of silent degradation. Observability is how you see it happening: trace every call (recording its inputs, output, and metadata) so you can inspect what the system actually did, sample its quality continuously after launch, and watch for drift in the inputs before it shows up as complaints. This chapter covers all three, and demonstrates drift detection on real data.

Setup for this chapter

Run in the gaba-core environment. The drift detection runs here. The tracing example uses Langfuse, which needs an account and keys, so it is shown without being executed during the book build.

from dotenv import load_dotenv
load_dotenv()

import matplotlib.pyplot as plt
import numpy as np

24.1 Tools in this chapter

Tool	Why we use it here	Alternatives	Trade-off
Langfuse, Arize Phoenix, LangSmith	record every model call’s inputs, output, cost, and latency for debugging and drift-watching	Helicone, OpenLLMetry (built on OpenTelemetry)	a hosted dashboard against self-hosting

The tooling-landscape appendix lists the current options for each.

24.2 Tracing every call

Because we cannot debug what we cannot observe, a trace records, for every model call, the inputs (prompt, retrieved context, tools), the output, and the metadata that matters: which model, how many tokens, what it cost, how long it took. With traces you can answer “why did the system give this user that answer,” reconstruct an incident, and identify the call that costs ten times the others. Tools like Langfuse, Arize Phoenix, and LangSmith (tracing and observability platforms for LLM calls) do this. The instrumentation is a thin wrapper around your model calls.

With an AI coding tool

Wiring the Langfuse client around a function like call_llm is boilerplate an assistant drafts quickly, and it is a safe place to let it: ask for the wrapped client and a decorator that tags each call with user and session. Then read what is actually recorded in the trace, since the trace table a few paragraphs below shows exactly what a wrapper like this records, and a default that logs the full untruncated prompt is a privacy decision you did not mean to make. Although the wrapping is routine, deciding what belongs in a stored trace requires your own judgment.

flowchart TB
    usr(["user request"]) --> ret["retrieve context"] --> mdl["model call"] --> ans(["answer"])
    ret -. "input,<br/>retrieved chunks" .-> ts[("trace store")]
    mdl -. "model + version, output,<br/>tokens, cost, latency" .-> ts
    ts --> ana["analyst queries<br/>and alerts"]

Figure 24.1: We trace every hop of a request. Each step writes its inputs, outputs, and metadata to the trace store, so when something goes wrong an analyst can replay exactly what happened, and alerts can fire on cost or latency without waiting for a complaint.

# With Langfuse (needs LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY in .env).
from langfuse.openai import openai  # drop-in wrapper that traces every call

client = openai.OpenAI(base_url="https://openrouter.ai/api/v1", api_key="...")
# Every call through this client is now traced: prompt, output, tokens, cost,
# latency, all visible in the Langfuse dashboard, grouped by user and session.
client.chat.completions.create(model="google/gemini-3.1-flash-lite",
                               messages=[{"role": "user", "content": "..."}])

The principle holds regardless of the specific tool: in production every call should be traced, because the first question when something goes wrong is always “show me exactly what happened,” and without traces you cannot answer it. A handful of traced steps from one request looks like this, in the form a dashboard displays:

Table 24.1: Five steps of one request, as a trace store records them. The fictional figures are illustrative, but the pattern is the one to watch for: the safety-check row costs more than ten times any other step, which is the kind of outlier that per-step tracing reveals and that an aggregate cost number averages away.

trace_id	step	model	tokens	cost	latency
t-9f2a	retrieve context	bge-m3 (embed)	180	$0.0000	40 ms
t-9f2a	classify intent	gemini-3.1-flash-lite	320	$0.0001	110 ms
t-9f2a	draft answer	gemini-3.1-flash-lite	1,240	$0.0004	520 ms
t-9f2a	rerank sources	bge-reranker-v2-m3	96	$0.0000	30 ms
t-9f2a	safety check	gpt-4.1-mini	2,980	$0.0048	900 ms

The trace format itself is standardizing: the OpenTelemetry GenAI semantic conventions (OpenTelemetry is the open standard for telemetry data), the 2025-26 direction the major platforms have converged on, define common attribute names for model, tokens, and cost, so traces from different tools can be collected in one backend.

24.3 Evaluating in production

Chapter 9 built an evaluation toolkit and ran it before launch. In production you keep running it, on a sample of live traffic. Judging every call can roughly double cost when the judge is as expensive as the primary model, but you can take a small percentage, run Chapter 9’s LLM-as-judge (a model scoring each answer against a rubric) over them, and track the faithfulness rate (how often the answer stays supported by its sources) or accuracy rate over time. A drop in that rate is an early warning, often the first sign that a model update or a prompt change degraded quality. The evaluation you built once stops being a launch gate and becomes a continuous monitor, sampled to keep its cost small. The observability platforms now run this loop natively, online LLM-judge evaluations over sampled traces, so in practice, with no loop of your own to build, you configure the judge and the sampling rate, though Chapter 9’s lesson stands: the judge is only as good as the rubric you give it.

Sampling small to keep cost down has a consequence that should inform the rate you set: a small sample is a noisy measurement, and noise hides a real drop for a while. We simulate ten weeks of weekly faithfulness judged on a sample of fifty, with a silent model update in week six that drops the true rate from 0.92 to 0.85, and watch how long the noise band keeps the drop invisible.

rng_mon = np.random.default_rng(7)
weeks = np.arange(1, 11)
n_sample = 50
true_rate = np.where(weeks < 6, 0.92, 0.85)   # silent update at week 6
observed = rng_mon.binomial(n_sample, true_rate) / n_sample
se = np.sqrt(observed * (1 - observed) / n_sample)   # binomial standard error

fig, ax = plt.subplots(figsize=(7.5, 4))
ax.step(weeks, true_rate, where="mid", color="#9a9a9a", linewidth=1.5,
        label="true rate")
ax.plot(weeks, observed, marker="o", color="#0969da", label="sampled estimate (n=50)")
ax.fill_between(weeks, observed - 1.96 * se, observed + 1.96 * se,
                color="#0969da", alpha=0.15, label="95% sampling band")
ax.axvline(6, color="#cf222e", linestyle=":", linewidth=1)
ax.text(6.1, ax.get_ylim()[0] + 0.01, "model update", fontsize=8, color="#cf222e")
ax.set_xlabel("week"); ax.set_ylabel("faithfulness rate")
ax.legend(frameon=False, fontsize=8)
ax.spines[["top", "right"]].set_visible(False)
plt.tight_layout(); plt.show()

Line chart over ten weeks: a gray step line for the true faithfulness rate dropping from 0.92 to 0.85 at week six, a blue line of noisy weekly sample estimates with a shaded band, the estimates wandering within the band for a couple of weeks after the drop before clearly falling below the original level. — Figure 24.2: Ten weeks of weekly faithfulness, each estimated from a 50-call judge sample with binomial noise. The true rate (gray) drops at the week-6 model update, but the sampled estimate stays inside its own noise band for about two weeks before the decline separates from chance, which is the detection delay a sample size buys.

The drop is real from week six, but the sampled line does not announce it: for the first week or two after the update the estimate sits inside the band it occupied before, because a seven-point fall is small next to the wobble of a fifty-call sample. Only once several post-update weeks accumulate does the decline clearly separate from chance. This lag is the reason sample size is a monitoring decision as well as a cost one: a larger weekly sample tightens the band and shortens the delay, a smaller one saves money and lengthens it, and where you set the rate is a deliberate trade between detection speed and judge cost.

24.4 Detecting drift before users do

The inputs themselves change over time, and a model tuned for last quarter’s questions can quietly underperform on this quarter’s. You can catch this without any labels by watching the distribution of inputs. Embed a reference batch from when the system was known to be healthy, and compare new traffic against it. If new inputs sit far from the reference distribution, something has shifted.

from gaba.embed import embed_texts
from gaba.data import load_tickets

tickets = load_tickets()["text"].tolist()
reference = tickets[:18]               # a healthy baseline
new_similar = tickets[18:28]           # later traffic, same kind of tickets

# A reference centroid: the average direction of healthy inputs.
ref_vectors = embed_texts(reference)
centroid = ref_vectors.mean(axis=0)
centroid /= np.linalg.norm(centroid)

def drift_score(batch: list[str]) -> float:
    """Mean cosine distance of a batch from the reference centroid."""
    vectors = embed_texts(batch)
    return float(np.mean([1 - (v @ centroid) / np.linalg.norm(v) for v in vectors]))

print(f"drift of similar new traffic: {drift_score(new_similar):.3f}")

drift of similar new traffic: 0.336

This score is the baseline drift, the measure of what normal variation looks like. Now compare a batch from a genuinely different distribution, standing in for the day your support inbox fills with a new kind of issue.

from datasets import load_dataset

other_domain = [r[:200] for r in
                load_dataset("stanfordnlp/imdb")["train"].shuffle(seed=0).select(range(10))["text"]]
print(f"drift of off-distribution traffic: {drift_score(other_domain):.3f}")

drift of off-distribution traffic: 0.506

The off-distribution batch has a clearly higher drift than the similar one. In production you set a threshold from the normal variation and alert when new traffic crosses it, which tells you the inputs have shifted before anyone files a complaint, while you still have time to re-evaluate and adapt.

This clean separation is also where the simple metric can mislead. Because real drift rarely arrives as a clean swap of the whole batch, the realistic case is a fraction of new traffic that has shifted, a new issue type appearing alongside business as usual. Consider what the centroid does with a batch that is only half shifted.

half_shifted = list(new_similar[:5]) + other_domain[:5]
print(f"half-shifted batch drift: {drift_score(half_shifted):.3f}")

half-shifted batch drift: 0.435

Half is one point on a curve. Sweeping the contaminated fraction from none to all and mixing in-distribution and off-distribution texts in each proportion, we draw the whole relationship between how much traffic has shifted and what the centroid metric reports.

fractions = [0.0, 0.25, 0.50, 0.75, 1.0]
pool_in, pool_out = list(new_similar), list(other_domain)
n_batch = 10
sweep_scores = []
rng = np.random.default_rng(0)
for frac in fractions:
    n_out = int(round(frac * n_batch))
    batch = ([pool_out[i % len(pool_out)] for i in range(n_out)]
             + [pool_in[i % len(pool_in)] for i in range(n_batch - n_out)])
    sweep_scores.append(drift_score(batch))

threshold = sweep_scores[-1] - 0.02   # an alert calibrated just under the full shift

fig, ax = plt.subplots(figsize=(6.5, 4))
ax.plot([f * 100 for f in fractions], sweep_scores, marker="o",
        color="#0969da", linewidth=2)
ax.axhline(threshold, color="#cf222e", linestyle="--", linewidth=1)
ax.text(2, threshold + 0.004, "alert threshold", fontsize=8, color="#cf222e")
missed = [f * 100 for f, s in zip(fractions, sweep_scores) if s < threshold]
if missed:
    ax.axvspan(0, max(missed), color="#bf8700", alpha=0.12)
ax.set_xlabel("off-distribution fraction of the batch (%)")
ax.set_ylabel("drift score (mean cosine distance)")
ax.spines[["top", "right"]].set_visible(False)
plt.tight_layout(); plt.show()

Line chart of drift score rising roughly linearly with the off-distribution fraction from 0 to 100 percent, with a horizontal dashed threshold line near the full-shift score and the region below it on the left shaded to mark missed mixtures. — Figure 24.3: Drift score as the off-distribution fraction of a batch rises from 0 to 100 percent. The score climbs almost linearly with contamination, so any alert threshold drawn as a horizontal line is missing every mixture whose score falls below it: the shaded span under a full-shift threshold is the range a mean-based detector lets through.

The curve rises with contamination, but it rises smoothly, which is the problem for any single threshold: set the alert just under the catastrophic full-shift score and the shaded band on the left is the whole range of partial shifts that pass underneath it, undetected, even though a quarter or half of the traffic has already turned over. The missed range follows from the metric’s design, which averages the shifted fraction against the calm one, so a finer measurement would not remove it. Lowering the threshold to catch the partial shifts trades the miss for false alarms on normal variation, which is the dilemma a mean-based detector cannot resolve and a distribution-level test can.

The three scores side by side make the dilution visible.

batch_names = ["similar\n(in-distribution)", "half-shifted", "different\n(off-distribution)"]
batch_scores = [drift_score(new_similar), drift_score(half_shifted),
                drift_score(other_domain)]

fig, ax = plt.subplots(figsize=(5.5, 3.5))
bars = ax.bar(batch_names, batch_scores,
              color=["#0969da", "#8250df", "#cf222e"], width=0.55)
ax.bar_label(bars, fmt="%.3f", padding=3)
ax.set_ylabel("drift score (mean cosine distance)")
ax.spines[["top", "right"]].set_visible(False)
plt.tight_layout()
plt.show()

Bar chart with three bars showing drift scores: about 0.34 for the in-distribution batch, about 0.44 for the half-shifted batch, and about 0.51 for the off-distribution batch. — Figure 24.4: We compare the drift scores of the three batches. The half-shifted batch falls roughly midway between the in-distribution baseline and the fully shifted batch, right at the mixture average, because the centroid mean averages the calm half together with the shifted half; a threshold calibrated near the full-shift score misses it.

The score falls roughly midway between the baseline and the fully shifted batch, which is exactly what averaging predicts: the batch is half calm and half shifted, so its mean distance settles near the average of the two regimes. Because the score settles there, a threshold calibrated near the full-shift score, the obvious catastrophe, does not flag this batch even though half the traffic has changed, which makes the averaging weakness concrete: a mean-based detector dilutes exactly the partial shift we most want to catch early. The metric is deliberately simple, the mean cosine distance (one minus the cosine similarity of two vectors, so larger means more different) to a reference centroid, and that simplicity is also its blind spot. Stronger two-sample tests (which compare two whole distributions for any difference), such as maximum mean discrepancy (a single number for how far apart two whole distributions sit) or a per-dimension comparison, look at the whole distribution, whereas the centroid summarizes only its center, and so catch what a centroid cannot. Even so, the simple version still serves as a first alarm, because it needs no labels, which makes it the cheapest early-warning system you can run, as long as you remember what it cannot see.

Don’t outsource this

Setting the alert threshold on drift_score is not a number an assistant should pick for you. The centroid metric averages a shifted fraction together with the calm majority, so a threshold tuned to catch the full-shift case is mathematically guaranteed to miss some partial shifts, and only you can decide how much of this blind spot your business can tolerate against the cost of false alarms. This trade belongs to the person who will answer the alerts, and no demo run can make it for them.

To see the blind spot directly, move the fraction of shifted traffic and the alert threshold, and observe where the alert fires and where it remains silent, noting that the expected score is the linear mixture of the two endpoints we just measured.

viewof drift_widget = {
  const lo = drift_lo, hi = drift_hi;
  const state = { frac: 0.5, thresh: +(hi - 0.02).toFixed(3) };

  const container = document.createElement("div");
  container.style.cssText = "font-family: inherit; color: var(--bs-body-color, #1f2328); border: 1px solid var(--bs-border-color, #d0d7de); border-radius: 6px; padding: 14px; max-width: 620px;";

  const slider = (key, label, min, max, step) => {
    const row = document.createElement("div");
    row.style.cssText = "display: flex; align-items: center; gap: 10px; margin-bottom: 8px; font-size: 13px;";
    const lab = document.createElement("span");
    lab.style.cssText = "flex: 1.3;";
    const s = document.createElement("input");
    s.type = "range"; s.min = min; s.max = max; s.step = step; s.value = state[key];
    s.style.cssText = "flex: 1;";
    s.oninput = () => { state[key] = +s.value; render(); };
    const set = () => { lab.textContent = label(state[key]); };
    s.addEventListener("input", set); set();
    row.append(lab, s);
    container.appendChild(row);
    return s;
  };

  slider("frac", v => `shifted fraction: ${Math.round(v * 100)}%`, 0, 1, 0.05);
  slider("thresh", v => `alert threshold: ${v.toFixed(3)}`, lo, hi, 0.001);

  const W = 560, H = 200, padL = 50, padB = 34, padT = 14;
  const svgNS = "http://www.w3.org/2000/svg";
  const x = f => padL + f * (W - padL - 12);
  const y = sc => padT + (1 - (sc - lo) / (hi - lo)) * (H - padT - padB);

  const readout = document.createElement("div");

  function render() {
    const expected = lo + state.frac * (hi - lo);
    const fires = expected >= state.thresh;
    // largest shifted fraction whose expected score is still below threshold
    const missedFrac = state.thresh <= lo ? 0 :
                       state.thresh >= hi ? 1 : (state.thresh - lo) / (hi - lo);

    const svg = document.createElementNS(svgNS, "svg");
    svg.setAttribute("viewBox", `0 0 ${W} ${H}`);
    svg.style.cssText = "width: 100%; height: auto;";

    // amber missed region: 0 to missedFrac
    if (missedFrac > 0) {
      const rect = document.createElementNS(svgNS, "rect");
      rect.setAttribute("x", x(0)); rect.setAttribute("y", padT);
      rect.setAttribute("width", x(missedFrac) - x(0));
      rect.setAttribute("height", H - padT - padB);
      rect.setAttribute("fill", "rgba(191,135,0,0.18)");
      svg.appendChild(rect);
    }

    // mixture line lo -> hi
    const line = document.createElementNS(svgNS, "line");
    line.setAttribute("x1", x(0)); line.setAttribute("y1", y(lo));
    line.setAttribute("x2", x(1)); line.setAttribute("y2", y(hi));
    line.style.stroke = "#0969da"; line.setAttribute("stroke-width", 2);
    svg.appendChild(line);

    // threshold line
    const th = document.createElementNS(svgNS, "line");
    th.setAttribute("x1", x(0)); th.setAttribute("y1", y(state.thresh));
    th.setAttribute("x2", x(1)); th.setAttribute("y2", y(state.thresh));
    th.style.stroke = "#cf222e"; th.setAttribute("stroke-width", 1.5);
    th.setAttribute("stroke-dasharray", "5 4");
    svg.appendChild(th);

    // current point
    const dot = document.createElementNS(svgNS, "circle");
    dot.setAttribute("cx", x(state.frac)); dot.setAttribute("cy", y(expected));
    dot.setAttribute("r", 6);
    dot.style.fill = fires ? "#cf222e" : "#bf8700";
    svg.appendChild(dot);

    // axis labels
    const xl = document.createElementNS(svgNS, "text");
    xl.setAttribute("x", (padL + W) / 2); xl.setAttribute("y", H - 6);
    xl.setAttribute("text-anchor", "middle"); xl.setAttribute("font-size", "11");
    xl.style.fill = "var(--bs-secondary-color, #57606a)";
    xl.textContent = "shifted fraction";
    svg.appendChild(xl);

    container.querySelectorAll("svg").forEach(s => s.remove());
    container.insertBefore(svg, readout);

    readout.style.cssText = "margin-top: 8px; font-size: 14px;";
    readout.innerHTML =
      `expected drift score <strong>${expected.toFixed(3)}</strong> ` +
      `vs threshold ${state.thresh.toFixed(3)}: ` +
      (fires
        ? `<span style="color:#cf222e;font-weight:600;">ALERT FIRES</span>`
        : `<span style="color:#bf8700;font-weight:600;">silent</span> ` +
          `(the shift is real but stays under the line)`);
  }

  container.appendChild(readout);
  render();
  return container;
}

24.5 Kill switches and graceful failure

The last piece of observability is what happens when monitoring fires. A production system needs a way to fail safe: a circuit breaker that stops calling a provider that is down (since a naive client would retry forever), a kill switch that can disable an agent’s risky tools without a redeploy, and a fallback to a simpler model or cached answer when the primary errors. Observability tells you something is wrong; these mechanisms keep “wrong” from becoming “harmful” while you fix it.

24.6 Evaluation: does drift detection catch a shift?

The drift detector is itself a system, and we check it the same way: does it flag a shifted batch and not flag a normal one?

Metric: the drift score, which should be clearly higher for off-distribution input than for in-distribution input.
Test set: a batch of similar new traffic and a batch from a different distribution.
Baseline: the normal drift of in-distribution traffic, which sets the alert threshold.

import pandas as pd
pd.DataFrame({
    "batch": ["similar (in-distribution)", "different (off-distribution)"],
    "drift score": [round(drift_score(new_similar), 3), round(drift_score(other_domain), 3)],
})

	batch	drift score
0	similar (in-distribution)	0.336
1	different (off-distribution)	0.506

The gap between the two is the signal. A real deployment computes the in-distribution drift over many healthy batches to learn its normal range, sets the alert threshold above that range, and watches. One caveat deserves emphasis: drift detection tells you the inputs changed, while whether quality dropped remains a separate question. The two usually go together, but the alert, which carries no verdict on its own, is a prompt to re-evaluate with the labeled tools of Chapter 9.

Compliance: traces are personal data

A trace captures the full prompt, and the full prompt contains whatever the customer wrote: names, account details, complaints, sometimes payment information. The trace store is therefore a store of personal data, and the retention, access, and residency rules of Appendix D apply to it just as they apply to the production database. Decide who can read traces, how long they are retained, and whether customer text needs masking before it is stored in a third-party observability platform.

Cost: observability is cheap insurance

Tracing adds negligible cost, a small logging overhead per call. Production evaluation costs a judge call on a small sample, a few percent of traffic. Drift detection is embeddings you may already be computing. On self-hosted hardware the same arithmetic appears as capacity: tracing adds negligible latency per call, and a sampled judge consumes only a small fraction of the throughput that judging every call would take from the same machines. Against these small ongoing costs sits the cost of a silent failure running for a month, which is the expensive thing observability exists to prevent.

24.7 Exercises

24.7.1 Conceptual questions

The defining failure of production LLM systems is that they:
1. Crash under load once traffic exceeds what testing covered
2. Leak data through verbose error messages in production logs
3. Grow more expensive each month as providers raise token prices
4. Degrade silently, so no one notices until users complain
A trace of a model call should record:
1. The inputs, output, and metadata: model, tokens, cost, and latency
2. Only the final answer, since the inputs may contain user data
3. The cost and latency alone, since the text is too large to store
4. A hash of the conversation, so incidents can be matched without storing text
Why judge only a sampled few percent of production traffic?
1. The judge model grows less accurate when it runs continuously
2. A random sample gives a more representative picture of quality than full coverage
3. Judging every call roughly doubles cost; a sample tracks quality cheaply
4. Providers rate-limit judge calls to a fixed fraction of traffic
Drift detection works without labels by:
1. Counting how often the model returns an error or an empty answer
2. Comparing the distribution of new inputs against a healthy reference batch
3. Asking a judge model to score a small sample of recent answers
4. Re-running the launch evaluation set each week and watching the accuracy trend
A drift alert tells you:
1. The inputs changed; the alert prompts a re-evaluation and carries no verdict on quality
2. Answer quality has dropped by an amount proportional to the rise in the score
3. The provider has silently updated the model behind the API
4. The embedding model is stale and needs to be replaced with a newer one
In the chapter’s demo, the half-shifted batch scored 0.435, roughly midway between the 0.336 baseline and the 0.506 full shift, so a threshold set near the full-shift score would miss it. Why does the score land there?
1. Half a batch is too small a sample for mean cosine distance to register the shift
2. The off-distribution texts were truncated to 200 characters, weakening their signal
3. The alert threshold was calibrated on the full shift, so the metric rescaled around it
4. The centroid mean averages the calm half with the shifted half, landing at the mixture average
Which approach would catch the partial shift that the centroid metric dilutes?
1. Lowering the alert threshold until even small score increases trigger an alert
2. A two-sample test such as maximum mean discrepancy, comparing whole distributions
3. Embedding both batches with a larger, more accurate embedding model
4. Averaging the drift scores over a longer window to smooth out batch noise
In evaluating the drift detector itself, what plays the role of the baseline?
1. The off-distribution batch, which sets the score the detector must reach
2. Zero drift, since identical batches should always produce a score of zero
3. The normal drift of in-distribution traffic, which sets the alert threshold
4. The faithfulness rate the judge model assigns to the same batches of traffic

24.7.2 Build lab

Compute the drift score over five separate in-distribution batches to estimate the normal range, then set an alert threshold (for example, the mean plus two standard deviations). Run the off-distribution batch and confirm it crosses the threshold while the in-distribution batches do not. Report your threshold and whether it cleanly separates the two.

24.7.3 Evaluate lab

This one is yours to design. Build a small production-evaluation loop: take ten answers from any earlier chapter’s system, judge them with the Chapter 9 judge, and record the faithfulness rate. Then imagine a model change degraded three of them, recompute, and report how large a sample you would need to detect that drop reliably. Decide what sampling rate you would run in production.

Where we go next

We can build capabilities, make them safe, and watch them in production. The last chapter assembles it: Chapter 25 wraps the accumulated retrieval-and-agent system in a small web service with a chat interface, the eval harness and observability built in, so a colleague can finally use what we built.

--- title: "Observability and drift" subtitle: "Watching a system that is live" jupyter: gaba-core --- A system that passed every test on the day it went live can be quietly wrong a month later. The model provider silently updates a model, the kinds of questions users ask shift, a prompt change three weeks ago degraded answers and no one noticed. Because production large language model (LLM) systems rarely break loudly, failure typically takes the form of silent degradation. Observability is how you see it happening: trace every call (recording its inputs, output, and metadata) so you can inspect what the system actually did, sample its quality continuously after launch, and watch for drift in the inputs before it shows up as complaints. This chapter covers all three, and demonstrates drift detection on real data. ::: {.callout-note title="Setup for this chapter"} Run in the `gaba-core` environment. The drift detection runs here. The tracing example uses Langfuse, which needs an account and keys, so it is shown without being executed during the book build. ::: ```{python} from dotenv import load_dotenv load_dotenv() import matplotlib.pyplot as plt import numpy as np ``` ## Tools in this chapter | Tool | Why we use it here | Alternatives | Trade-off | |---|---|---|---| | **Langfuse, Arize Phoenix, LangSmith** | record every model call's inputs, output, cost, and latency for debugging and drift-watching | Helicone, OpenLLMetry (built on OpenTelemetry) | a hosted dashboard against self-hosting | The [tooling-landscape appendix](../../appendices/i-tooling.qmd) lists the current options for each. ## Tracing every call Because we cannot debug what we cannot observe, a trace records, for every model call, the inputs (prompt, retrieved context, tools), the output, and the metadata that matters: which model, how many tokens, what it cost, how long it took. With traces you can answer "why did the system give this user that answer," reconstruct an incident, and identify the call that costs ten times the others. Tools like Langfuse, Arize Phoenix, and LangSmith (tracing and observability platforms for LLM calls) do this. The instrumentation is a thin wrapper around your model calls. ::: {.callout-tip title="With an AI coding tool"} Wiring the Langfuse client around a function like `call_llm` is boilerplate an assistant drafts quickly, and it is a safe place to let it: ask for the wrapped client and a decorator that tags each call with user and session. Then read what is actually recorded in the trace, since the trace table a few paragraphs below shows exactly what a wrapper like this records, and a default that logs the full untruncated prompt is a privacy decision you did not mean to make. Although the wrapping is routine, deciding what belongs in a stored trace requires your own judgment. ::: ```{mermaid} %%| echo: false %%| label: fig-tracing %%| fig-cap: "We trace every hop of a request. Each step writes its inputs, outputs, and metadata to the trace store, so when something goes wrong an analyst can replay exactly what happened, and alerts can fire on cost or latency without waiting for a complaint." flowchart TB usr(["user request"]) --> ret["retrieve context"] --> mdl["model call"] --> ans(["answer"]) ret -. "input, retrieved chunks" .-> ts[("trace store")] mdl -. "model + version, output, tokens, cost, latency" .-> ts ts --> ana["analyst queries and alerts"] ``` ```{python} #| eval: false # With Langfuse (needs LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY in .env). from langfuse.openai import openai # drop-in wrapper that traces every call client = openai.OpenAI(base_url="https://openrouter.ai/api/v1", api_key="...") # Every call through this client is now traced: prompt, output, tokens, cost, # latency, all visible in the Langfuse dashboard, grouped by user and session. client.chat.completions.create(model="google/gemini-3.1-flash-lite", messages=[{"role": "user", "content": "..."}]) ``` The principle holds regardless of the specific tool: in production every call should be traced, because the first question when something goes wrong is always "show me exactly what happened," and without traces you cannot answer it. A handful of traced steps from one request looks like this, in the form a dashboard displays: | trace_id | step | model | tokens | cost | latency | |:---------|:-----|:------|-------:|-----:|--------:| | t-9f2a | retrieve context | bge-m3 (embed) | 180 | $0.0000 | 40 ms | | t-9f2a | classify intent | gemini-3.1-flash-lite | 320 | $0.0001 | 110 ms | | t-9f2a | draft answer | gemini-3.1-flash-lite | 1,240 | $0.0004 | 520 ms | | t-9f2a | rerank sources | bge-reranker-v2-m3 | 96 | $0.0000 | 30 ms | | t-9f2a | safety check | gpt-4.1-mini | 2,980 | $0.0048 | 900 ms | : Five steps of one request, as a trace store records them. The fictional figures are illustrative, but the pattern is the one to watch for: the safety-check row costs more than ten times any other step, which is the kind of outlier that per-step tracing reveals and that an aggregate cost number averages away. {#tbl-trace} The trace format itself is standardizing: the OpenTelemetry GenAI semantic conventions (OpenTelemetry is the open standard for telemetry data), the 2025-26 direction the major platforms have converged on, define common attribute names for model, tokens, and cost, so traces from different tools can be collected in one backend. ## Evaluating in production Chapter 9 built an evaluation toolkit and ran it before launch. In production you keep running it, on a sample of live traffic. Judging every call can roughly double cost when the judge is as expensive as the primary model, but you can take a small percentage, run Chapter 9's LLM-as-judge (a model scoring each answer against a rubric) over them, and track the faithfulness rate (how often the answer stays supported by its sources) or accuracy rate over time. A drop in that rate is an early warning, often the first sign that a model update or a prompt change degraded quality. The evaluation you built once stops being a launch gate and becomes a continuous monitor, sampled to keep its cost small. The observability platforms now run this loop natively, online LLM-judge evaluations over sampled traces, so in practice, with no loop of your own to build, you configure the judge and the sampling rate, though Chapter 9's lesson stands: the judge is only as good as the rubric you give it. Sampling small to keep cost down has a consequence that should inform the rate you set: a small sample is a noisy measurement, and noise hides a real drop for a while. We simulate ten weeks of weekly faithfulness judged on a sample of fifty, with a silent model update in week six that drops the true rate from 0.92 to 0.85, and watch how long the noise band keeps the drop invisible. ```{python} #| label: fig-prod-monitor #| fig-cap: "Ten weeks of weekly faithfulness, each estimated from a 50-call judge sample with binomial noise. The true rate (gray) drops at the week-6 model update, but the sampled estimate stays inside its own noise band for about two weeks before the decline separates from chance, which is the detection delay a sample size buys." #| fig-alt: "Line chart over ten weeks: a gray step line for the true faithfulness rate dropping from 0.92 to 0.85 at week six, a blue line of noisy weekly sample estimates with a shaded band, the estimates wandering within the band for a couple of weeks after the drop before clearly falling below the original level." rng_mon = np.random.default_rng(7) weeks = np.arange(1, 11) n_sample = 50 true_rate = np.where(weeks < 6, 0.92, 0.85) # silent update at week 6 observed = rng_mon.binomial(n_sample, true_rate) / n_sample se = np.sqrt(observed * (1 - observed) / n_sample) # binomial standard error fig, ax = plt.subplots(figsize=(7.5, 4)) ax.step(weeks, true_rate, where="mid", color="#9a9a9a", linewidth=1.5, label="true rate") ax.plot(weeks, observed, marker="o", color="#0969da", label="sampled estimate (n=50)") ax.fill_between(weeks, observed - 1.96 * se, observed + 1.96 * se, color="#0969da", alpha=0.15, label="95% sampling band") ax.axvline(6, color="#cf222e", linestyle=":", linewidth=1) ax.text(6.1, ax.get_ylim()[0] + 0.01, "model update", fontsize=8, color="#cf222e") ax.set_xlabel("week"); ax.set_ylabel("faithfulness rate") ax.legend(frameon=False, fontsize=8) ax.spines[["top", "right"]].set_visible(False) plt.tight_layout(); plt.show() ``` The drop is real from week six, but the sampled line does not announce it: for the first week or two after the update the estimate sits inside the band it occupied before, because a seven-point fall is small next to the wobble of a fifty-call sample. Only once several post-update weeks accumulate does the decline clearly separate from chance. This lag is the reason sample size is a monitoring decision as well as a cost one: a larger weekly sample tightens the band and shortens the delay, a smaller one saves money and lengthens it, and where you set the rate is a deliberate trade between detection speed and judge cost. ## Detecting drift before users do The inputs themselves change over time, and a model tuned for last quarter's questions can quietly underperform on this quarter's. You can catch this without any labels by watching the *distribution* of inputs. Embed a reference batch from when the system was known to be healthy, and compare new traffic against it. If new inputs sit far from the reference distribution, something has shifted. ```{python} from gaba.embed import embed_texts from gaba.data import load_tickets tickets = load_tickets()["text"].tolist() reference = tickets[:18] # a healthy baseline new_similar = tickets[18:28] # later traffic, same kind of tickets # A reference centroid: the average direction of healthy inputs. ref_vectors = embed_texts(reference) centroid = ref_vectors.mean(axis=0) centroid /= np.linalg.norm(centroid) def drift_score(batch: list[str]) -> float: """Mean cosine distance of a batch from the reference centroid.""" vectors = embed_texts(batch) return float(np.mean([1 - (v @ centroid) / np.linalg.norm(v) for v in vectors])) print(f"drift of similar new traffic: {drift_score(new_similar):.3f}") ``` This score is the baseline drift, the measure of what normal variation looks like. Now compare a batch from a genuinely different distribution, standing in for the day your support inbox fills with a new kind of issue. ```{python} from datasets import load_dataset other_domain = [r[:200] for r in load_dataset("stanfordnlp/imdb")["train"].shuffle(seed=0).select(range(10))["text"]] print(f"drift of off-distribution traffic: {drift_score(other_domain):.3f}") ``` The off-distribution batch has a clearly higher drift than the similar one. In production you set a threshold from the normal variation and alert when new traffic crosses it, which tells you the inputs have shifted before anyone files a complaint, while you still have time to re-evaluate and adapt. This clean separation is also where the simple metric can mislead. Because real drift rarely arrives as a clean swap of the whole batch, the realistic case is a *fraction* of new traffic that has shifted, a new issue type appearing alongside business as usual. Consider what the centroid does with a batch that is only half shifted. ```{python} half_shifted = list(new_similar[:5]) + other_domain[:5] print(f"half-shifted batch drift: {drift_score(half_shifted):.3f}") ``` Half is one point on a curve. Sweeping the contaminated fraction from none to all and mixing in-distribution and off-distribution texts in each proportion, we draw the whole relationship between how much traffic has shifted and what the centroid metric reports. ```{python} #| label: fig-contamination-sweep #| fig-cap: "Drift score as the off-distribution fraction of a batch rises from 0 to 100 percent. The score climbs almost linearly with contamination, so any alert threshold drawn as a horizontal line is missing every mixture whose score falls below it: the shaded span under a full-shift threshold is the range a mean-based detector lets through." #| fig-alt: "Line chart of drift score rising roughly linearly with the off-distribution fraction from 0 to 100 percent, with a horizontal dashed threshold line near the full-shift score and the region below it on the left shaded to mark missed mixtures." fractions = [0.0, 0.25, 0.50, 0.75, 1.0] pool_in, pool_out = list(new_similar), list(other_domain) n_batch = 10 sweep_scores = [] rng = np.random.default_rng(0) for frac in fractions: n_out = int(round(frac * n_batch)) batch = ([pool_out[i % len(pool_out)] for i in range(n_out)] + [pool_in[i % len(pool_in)] for i in range(n_batch - n_out)]) sweep_scores.append(drift_score(batch)) threshold = sweep_scores[-1] - 0.02 # an alert calibrated just under the full shift fig, ax = plt.subplots(figsize=(6.5, 4)) ax.plot([f * 100 for f in fractions], sweep_scores, marker="o", color="#0969da", linewidth=2) ax.axhline(threshold, color="#cf222e", linestyle="--", linewidth=1) ax.text(2, threshold + 0.004, "alert threshold", fontsize=8, color="#cf222e") missed = [f * 100 for f, s in zip(fractions, sweep_scores) if s < threshold] if missed: ax.axvspan(0, max(missed), color="#bf8700", alpha=0.12) ax.set_xlabel("off-distribution fraction of the batch (%)") ax.set_ylabel("drift score (mean cosine distance)") ax.spines[["top", "right"]].set_visible(False) plt.tight_layout(); plt.show() ``` The curve rises with contamination, but it rises *smoothly*, which is the problem for any single threshold: set the alert just under the catastrophic full-shift score and the shaded band on the left is the whole range of partial shifts that pass underneath it, undetected, even though a quarter or half of the traffic has already turned over. The missed range follows from the metric's design, which averages the shifted fraction against the calm one, so a finer measurement would not remove it. Lowering the threshold to catch the partial shifts trades the miss for false alarms on normal variation, which is the dilemma a mean-based detector cannot resolve and a distribution-level test can. The three scores side by side make the dilution visible. ```{python} #| label: fig-drift-scores #| fig-cap: "We compare the drift scores of the three batches. The half-shifted batch falls roughly midway between the in-distribution baseline and the fully shifted batch, right at the mixture average, because the centroid mean averages the calm half together with the shifted half; a threshold calibrated near the full-shift score misses it." #| fig-alt: "Bar chart with three bars showing drift scores: about 0.34 for the in-distribution batch, about 0.44 for the half-shifted batch, and about 0.51 for the off-distribution batch." batch_names = ["similar\n(in-distribution)", "half-shifted", "different\n(off-distribution)"] batch_scores = [drift_score(new_similar), drift_score(half_shifted), drift_score(other_domain)] fig, ax = plt.subplots(figsize=(5.5, 3.5)) bars = ax.bar(batch_names, batch_scores, color=["#0969da", "#8250df", "#cf222e"], width=0.55) ax.bar_label(bars, fmt="%.3f", padding=3) ax.set_ylabel("drift score (mean cosine distance)") ax.spines[["top", "right"]].set_visible(False) plt.tight_layout() plt.show() ``` The score falls roughly midway between the baseline and the fully shifted batch, which is exactly what averaging predicts: the batch is half calm and half shifted, so its mean distance settles near the average of the two regimes. Because the score settles there, a threshold calibrated near the full-shift score, the obvious catastrophe, does not flag this batch even though half the traffic has changed, which makes the averaging weakness concrete: a mean-based detector dilutes exactly the partial shift we most want to catch early. The metric is deliberately simple, the mean cosine distance (one minus the cosine similarity of two vectors, so larger means more different) to a reference centroid, and that simplicity is also its blind spot. Stronger two-sample tests (which compare two whole distributions for any difference), such as maximum mean discrepancy (a single number for how far apart two whole distributions sit) or a per-dimension comparison, look at the whole distribution, whereas the centroid summarizes only its center, and so catch what a centroid cannot. Even so, the simple version still serves as a first alarm, because it needs no labels, which makes it the cheapest early-warning system you can run, as long as you remember what it cannot see. ::: {.callout-warning title="Don't outsource this"} Setting the alert threshold on `drift_score` is not a number an assistant should pick for you. The centroid metric averages a shifted fraction together with the calm majority, so a threshold tuned to catch the full-shift case is mathematically guaranteed to miss some partial shifts, and only you can decide how much of this blind spot your business can tolerate against the cost of false alarms. This trade belongs to the person who will answer the alerts, and no demo run can make it for them. ::: ```{python} #| echo: false # Hand the measured endpoints to the threshold widget below (online edition only). ojs_define(drift_lo=round(sweep_scores[0], 4), drift_hi=round(sweep_scores[-1], 4)) ``` ::: {.content-visible when-format="html"} To see the blind spot directly, move the fraction of shifted traffic and the alert threshold, and observe where the alert fires and where it remains silent, noting that the expected score is the linear mixture of the two endpoints we just measured. ```{ojs} //| echo: false //| label: fig-drift-threshold-widget //| fig-cap: "A drift alert as two dials. The expected score interpolates the chapter's measured endpoints by the shifted fraction; the alert fires only when that score crosses the threshold, and the amber region is every contamination level that has shifted but stays below the line, silent." viewof drift_widget = { const lo = drift_lo, hi = drift_hi; const state = { frac: 0.5, thresh: +(hi - 0.02).toFixed(3) }; const container = document.createElement("div"); container.style.cssText = "font-family: inherit; color: var(--bs-body-color, #1f2328); border: 1px solid var(--bs-border-color, #d0d7de); border-radius: 6px; padding: 14px; max-width: 620px;"; const slider = (key, label, min, max, step) => { const row = document.createElement("div"); row.style.cssText = "display: flex; align-items: center; gap: 10px; margin-bottom: 8px; font-size: 13px;"; const lab = document.createElement("span"); lab.style.cssText = "flex: 1.3;"; const s = document.createElement("input"); s.type = "range"; s.min = min; s.max = max; s.step = step; s.value = state[key]; s.style.cssText = "flex: 1;"; s.oninput = () => { state[key] = +s.value; render(); }; const set = () => { lab.textContent = label(state[key]); }; s.addEventListener("input", set); set(); row.append(lab, s); container.appendChild(row); return s; }; slider("frac", v => `shifted fraction: ${Math.round(v * 100)}%`, 0, 1, 0.05); slider("thresh", v => `alert threshold: ${v.toFixed(3)}`, lo, hi, 0.001); const W = 560, H = 200, padL = 50, padB = 34, padT = 14; const svgNS = "http://www.w3.org/2000/svg"; const x = f => padL + f * (W - padL - 12); const y = sc => padT + (1 - (sc - lo) / (hi - lo)) * (H - padT - padB); const readout = document.createElement("div"); function render() { const expected = lo + state.frac * (hi - lo); const fires = expected >= state.thresh; // largest shifted fraction whose expected score is still below threshold const missedFrac = state.thresh <= lo ? 0 : state.thresh >= hi ? 1 : (state.thresh - lo) / (hi - lo); const svg = document.createElementNS(svgNS, "svg"); svg.setAttribute("viewBox", `0 0 ${W} ${H}`); svg.style.cssText = "width: 100%; height: auto;"; // amber missed region: 0 to missedFrac if (missedFrac > 0) { const rect = document.createElementNS(svgNS, "rect"); rect.setAttribute("x", x(0)); rect.setAttribute("y", padT); rect.setAttribute("width", x(missedFrac) - x(0)); rect.setAttribute("height", H - padT - padB); rect.setAttribute("fill", "rgba(191,135,0,0.18)"); svg.appendChild(rect); } // mixture line lo -> hi const line = document.createElementNS(svgNS, "line"); line.setAttribute("x1", x(0)); line.setAttribute("y1", y(lo)); line.setAttribute("x2", x(1)); line.setAttribute("y2", y(hi)); line.style.stroke = "#0969da"; line.setAttribute("stroke-width", 2); svg.appendChild(line); // threshold line const th = document.createElementNS(svgNS, "line"); th.setAttribute("x1", x(0)); th.setAttribute("y1", y(state.thresh)); th.setAttribute("x2", x(1)); th.setAttribute("y2", y(state.thresh)); th.style.stroke = "#cf222e"; th.setAttribute("stroke-width", 1.5); th.setAttribute("stroke-dasharray", "5 4"); svg.appendChild(th); // current point const dot = document.createElementNS(svgNS, "circle"); dot.setAttribute("cx", x(state.frac)); dot.setAttribute("cy", y(expected)); dot.setAttribute("r", 6); dot.style.fill = fires ? "#cf222e" : "#bf8700"; svg.appendChild(dot); // axis labels const xl = document.createElementNS(svgNS, "text"); xl.setAttribute("x", (padL + W) / 2); xl.setAttribute("y", H - 6); xl.setAttribute("text-anchor", "middle"); xl.setAttribute("font-size", "11"); xl.style.fill = "var(--bs-secondary-color, #57606a)"; xl.textContent = "shifted fraction"; svg.appendChild(xl); container.querySelectorAll("svg").forEach(s => s.remove()); container.insertBefore(svg, readout); readout.style.cssText = "margin-top: 8px; font-size: 14px;"; readout.innerHTML = `expected drift score ${expected.toFixed(3)} ` + `vs threshold ${state.thresh.toFixed(3)}: ` + (fires ? `ALERT FIRES` : `silent ` + `(the shift is real but stays under the line)`); } container.appendChild(readout); render(); return container; } ``` ::: ::: {.content-visible when-format="pdf,epub"} The online edition turns this into two dials, shifted fraction and alert threshold, that show exactly where the alert fires and where it stays silent. The bar chart above is the static version: the half-shifted bar is the point a threshold set near the full-shift bar misses. ::: ## Kill switches and graceful failure The last piece of observability is what happens when monitoring fires. A production system needs a way to fail safe: a circuit breaker that stops calling a provider that is down (since a naive client would retry forever), a kill switch that can disable an agent's risky tools without a redeploy, and a fallback to a simpler model or cached answer when the primary errors. Observability tells you something is wrong; these mechanisms keep "wrong" from becoming "harmful" while you fix it. ## Evaluation: does drift detection catch a shift? The drift detector is itself a system, and we check it the same way: does it flag a shifted batch and not flag a normal one? **Metric**: the drift score, which should be clearly higher for off-distribution input than for in-distribution input.\ **Test set**: a batch of similar new traffic and a batch from a different distribution.\ **Baseline**: the normal drift of in-distribution traffic, which sets the alert threshold. ```{python} import pandas as pd pd.DataFrame({ "batch": ["similar (in-distribution)", "different (off-distribution)"], "drift score": [round(drift_score(new_similar), 3), round(drift_score(other_domain), 3)], }) ``` The gap between the two is the signal. A real deployment computes the in-distribution drift over many healthy batches to learn its normal range, sets the alert threshold above that range, and watches. One caveat deserves emphasis: drift detection tells you the inputs *changed*, while whether quality *dropped* remains a separate question. The two usually go together, but the alert, which carries no verdict on its own, is a prompt to re-evaluate with the labeled tools of Chapter 9. ::: {.callout-important title="Compliance: traces are personal data"} A trace captures the full prompt, and the full prompt contains whatever the customer wrote: names, account details, complaints, sometimes payment information. The trace store is therefore a store of personal data, and the retention, access, and residency rules of Appendix D apply to it just as they apply to the production database. Decide who can read traces, how long they are retained, and whether customer text needs masking before it is stored in a third-party observability platform. ::: ::: {.callout-tip title="Cost: observability is cheap insurance"} Tracing adds negligible cost, a small logging overhead per call. Production evaluation costs a judge call on a small sample, a few percent of traffic. Drift detection is embeddings you may already be computing. On self-hosted hardware the same arithmetic appears as capacity: tracing adds negligible latency per call, and a sampled judge consumes only a small fraction of the throughput that judging every call would take from the same machines. Against these small ongoing costs sits the cost of a silent failure running for a month, which is the expensive thing observability exists to prevent. ::: ## Exercises ### Conceptual questions 1. The defining failure of production LLM systems is that they: A. Crash under load once traffic exceeds what testing covered B. Leak data through verbose error messages in production logs C. Grow more expensive each month as providers raise token prices D. Degrade silently, so no one notices until users complain 2. A trace of a model call should record: A. The inputs, output, and metadata: model, tokens, cost, and latency B. Only the final answer, since the inputs may contain user data C. The cost and latency alone, since the text is too large to store D. A hash of the conversation, so incidents can be matched without storing text 3. Why judge only a sampled few percent of production traffic? A. The judge model grows less accurate when it runs continuously B. A random sample gives a more representative picture of quality than full coverage C. Judging every call roughly doubles cost; a sample tracks quality cheaply D. Providers rate-limit judge calls to a fixed fraction of traffic 4. Drift detection works without labels by: A. Counting how often the model returns an error or an empty answer B. Comparing the distribution of new inputs against a healthy reference batch C. Asking a judge model to score a small sample of recent answers D. Re-running the launch evaluation set each week and watching the accuracy trend 5. A drift alert tells you: A. The inputs changed; the alert prompts a re-evaluation and carries no verdict on quality B. Answer quality has dropped by an amount proportional to the rise in the score C. The provider has silently updated the model behind the API D. The embedding model is stale and needs to be replaced with a newer one 6. In the chapter's demo, the half-shifted batch scored 0.435, roughly midway between the 0.336 baseline and the 0.506 full shift, so a threshold set near the full-shift score would miss it. Why does the score land there? A. Half a batch is too small a sample for mean cosine distance to register the shift B. The off-distribution texts were truncated to 200 characters, weakening their signal C. The alert threshold was calibrated on the full shift, so the metric rescaled around it D. The centroid mean averages the calm half with the shifted half, landing at the mixture average 7. Which approach would catch the partial shift that the centroid metric dilutes? A. Lowering the alert threshold until even small score increases trigger an alert B. A two-sample test such as maximum mean discrepancy, comparing whole distributions C. Embedding both batches with a larger, more accurate embedding model D. Averaging the drift scores over a longer window to smooth out batch noise 8. In evaluating the drift detector itself, what plays the role of the baseline? A. The off-distribution batch, which sets the score the detector must reach B. Zero drift, since identical batches should always produce a score of zero C. The normal drift of in-distribution traffic, which sets the alert threshold D. The faithfulness rate the judge model assigns to the same batches of traffic ### Build lab Compute the drift score over five separate in-distribution batches to estimate the normal range, then set an alert threshold (for example, the mean plus two standard deviations). Run the off-distribution batch and confirm it crosses the threshold while the in-distribution batches do not. Report your threshold and whether it cleanly separates the two. ### Evaluate lab This one is yours to design. Build a small production-evaluation loop: take ten answers from any earlier chapter's system, judge them with the Chapter 9 judge, and record the faithfulness rate. Then imagine a model change degraded three of them, recompute, and report how large a sample you would need to detect that drop reliably. Decide what sampling rate you would run in production. ::: {.callout-note title="Where we go next"} We can build capabilities, make them safe, and watch them in production. The last chapter assembles it: Chapter 25 wraps the accumulated retrieval-and-agent system in a small web service with a chat interface, the eval harness and observability built in, so a colleague can finally use what we built. :::