from dotenv import load_dotenv
load_dotenv()True25 From notebook to shipped product
Wrapping the system in a service a colleague can use
Everything in this book has run in a notebook. A notebook is where you build and measure, although a colleague in another department cannot get an answer from one. The last step is to wrap what we have accumulated in a small service with a chat interface, so someone who has never opened Python can use it. And what we have accumulated is more than one function: grounded document answering (Chapter 7), the ticket triage that has run through this book since Chapter 2, and an agent that composes tools when a question has steps (Chapter 12). Each arrives with the disciplines it was built under: the typed schema of Chapter 3, the unit-cost accounting of Chapter 2, and the guardrails of Chapter 23. By the time it goes live, each function has already been measured and defended. This chapter assembles all three into one deployable application and, just as important, lays out what has to be deployed with it, the eval harness, the tracing, and a clear plan for whether the pilot should graduate to production at all, because most do not.
NoteSetup for this chapter
Run in the gaba-core environment with an OPENROUTER_API_KEY for the assembled-system check. The web service and chat UI (user interface) use the gaba-prod environment (FastAPI for the web service, Chainlit for the chat interface). This code is shown but not run during the book build, since a server is a long-running process that cannot complete in a notebook cell.
25.1 The system we already have
The work here is assembly, because every piece already exists, and there are three pieces. First, grounded document answering:
from gaba.rag import rag_answer
answer, sources = rag_answer("How did the AWS cloud segment perform?")
print(answer[:200])
print("\nsources:", sorted({s["ticker"] for s in sources}))In early 2023, AWS experienced substantial cost optimization as companies sought to reduce expenses in an uncertain economy. This was partly due to AWS assisting customers in using the cloud more effi
sources: ['AMZN']Second, the ticket triage this book opened with, now in its Chapter 3 form: a structured, validated object ready for a queue.
from typing import Literal
from pydantic import BaseModel
from gaba.llm import call_structured
class Triage(BaseModel):
category: Literal["billing", "technical", "shipping", "account", "other"]
urgency: Literal["low", "medium", "high"]
summary: str
ticket = "I was charged twice for my March order and nobody has replied for a week."
print(call_structured(ticket, Triage, system="Triage this support ticket.").data)category='billing' urgency='high' summary='Customer was charged twice for their March order and has not received a response for a week.'Third, the agent, for the questions that have steps. We give it the document search and a calculator, the Chapter 12 recipe:
from gaba.agent import run_agent
from gaba.rag import retrieve
def search_reports(query: str):
# Search the annual-report corpus.
return " | ".join(f"[{h['ticker']}] {h['text']}" for h in retrieve(query, n=3))
def calculator(expression: str):
# Evaluate arithmetic. Demo only; use a real parser in production (Ch 12).
return eval(expression, {"__builtins__": {}}, {})
TOOLS = {"search_reports": search_reports, "calculator": calculator}
SCHEMAS = [
{"type": "function", "function": {"name": "search_reports",
"description": "Search company annual reports for a passage.",
"parameters": {"type": "object", "properties": {"query": {"type": "string"}},
"required": ["query"]}}},
{"type": "function", "function": {"name": "calculator",
"description": "Evaluate an arithmetic expression.",
"parameters": {"type": "object", "properties": {"expression": {"type": "string"}},
"required": ["expression"]}}},
]
result = run_agent(
"What is roughly one tenth of AWS's 2023 net sales, in billions of dollars?",
tools=TOOLS, schemas=SCHEMAS,
system="You are a financial analyst assistant. Use the tools.",
)
print(result.answer)One tenth of AWS's 2023 net sales is roughly 9.1 billion dollars.These three functions are the product; everything from here is packaging, placing them behind an interface that other people can reach.
25.2 A web service
The first wrapper is an HTTP (HyperText Transfer Protocol) endpoint, a single URL the service answers requests at. FastAPI (a Python web framework for building APIs) turns our answer function into an API (application programming interface) that any application, be it a website, an internal tool, or a mobile app, can call.
Defining the app is ordinary Python, so we do it here for real. Only the serving step (uvicorn app:app, the server that runs the app) is a long-running process that runs outside a notebook.
TipWith an AI coding tool
Scaffolding app.py, the Query and TicketIn models, and the three route decorators is routine enough to hand to an assistant, and doing so is a fast way to get the structure right. Read every field it produces against the schemas the chapters already validated: does /triage actually return the Triage model’s fields, does /analyst expose the tool trace or only the final answer, and does a malformed request get rejected before it reaches a model. The generated wiring is trustworthy only once you have confirmed that it defers to the typed models you already built, because an assistant will sometimes invent a parallel set of its own.
# app.py -- serve with: uvicorn app:app
from fastapi import FastAPI
from pydantic import BaseModel
from gaba.rag import rag_answer
app = FastAPI(title="Analytics Assistant")
class Query(BaseModel):
question: str
class TicketIn(BaseModel):
text: str
@app.post("/ask")
def ask(query: Query):
answer, sources = rag_answer(query.question)
return {"answer": answer, "sources": sorted({s["ticker"] for s in sources})}
@app.post("/triage")
def triage(ticket: TicketIn):
return call_structured(ticket.text, Triage,
system="Triage this support ticket.").data.model_dump()
@app.post("/analyst")
def analyst(query: Query):
result = run_agent(query.question, tools=TOOLS, schemas=SCHEMAS,
system="You are a financial analyst assistant. Use the tools.")
return {"answer": result.answer, "steps": [st["tool"] for st in result.steps]}This is a complete API: documents, tickets, and multi-step questions, each behind its own endpoint. Each endpoint’s request is typed with a Pydantic model (Chapter 3) so malformed requests are rejected before they reach a model. And because the app is a real object, we can prove all of this right now: FastAPI’s test client serves it in-process and speaks actual HTTP, the same request handling uvicorn would run.
import time
from fastapi.testclient import TestClient
client = TestClient(app)
timings = {}
start = time.perf_counter()
r = client.post("/ask", json={"question": "How did the AWS cloud segment perform?"})
timings["/ask"] = (r.status_code, time.perf_counter() - start)
print(r.status_code, "| sources:", r.json()["sources"])
start = time.perf_counter()
r = client.post("/triage", json={"text": ticket})
timings["/triage"] = (r.status_code, time.perf_counter() - start)
print(r.status_code, "|", r.json())200 | sources: ['AMZN']
200 | {'category': 'billing', 'urgency': 'high', 'summary': 'Customer was charged twice for their March order and has not received a response for a week.'}start = time.perf_counter()
r = client.post("/analyst", json={
"question": "What is roughly one tenth of AWS's 2023 net sales, in billions of dollars?"})
timings["/analyst"] = (r.status_code, time.perf_counter() - start)
print(r.status_code, "| steps:", r.json()["steps"])
print(r.json()["answer"][:160])200 | steps: ['search_reports', 'calculator']
Roughly one tenth of AWS's 2023 net sales is 9.1 billion dollars.Because we timed each request, the service’s latency profile is already a table:
import pandas as pd
pd.DataFrame([{"endpoint": e, "status": s, "latency (s)": f"{t:.1f}"}
for e, (s, t) in timings.items()])| endpoint | status | latency (s) | |
|---|---|---|---|
| 0 | /ask | 200 | 1.4 |
| 1 | /triage | 200 | 0.8 |
| 2 | /analyst | 200 | 2.5 |
The agent endpoint is the slow one, and predictably so: an agent run is several model calls in sequence (Chapter 12’s loop), so its latency is the sum of its steps, which is something to set user expectations around before putting it behind a chat box.
With three endpoints answering live over HTTP, the typed models can show their value on the unhappy path, which we can also prove: a malformed request never reaches a model, because it is rejected immediately with a validation error.
bad = client.post("/ask", json={"q": "wrong field name"})
print(bad.status_code, "|", bad.json()["detail"][0]["msg"])422 | Field requiredThis 422, the HTTP status for an unprocessable request, is Chapter 3’s loud-failure principle operating at the service boundary. The error is immediate, structured, and free, whereas an unvalidated request would produce a confused model answer that costs tokens and trust.
Sources travel with every /ask answer so a caller can always check the grounding. /analyst returns the tool trace alongside the answer, because an agent’s path is part of its output (Chapter 12).
25.3 A chat interface
Because few business users will call an API directly, the second wrapper is a chat box. Chainlit turns the same function into a chat application in a few lines of code, with history and a place to display sources. (Chainlit is one choice among several; Gradio, Streamlit, and Open WebUI wrap a function just as readily.)
# chat.py -- run with: chainlit run chat.py
import chainlit as cl
from gaba.rag import rag_answer
@cl.on_message
async def main(message: cl.Message):
answer, sources = rag_answer(message.content)
tickers = sorted({s["ticker"] for s in sources})
await cl.Message(content=f"{answer}\n\n*Sources: {', '.join(tickers)}*").send()A colleague opens a web page, types a question, and gets a grounded answer with its sources. The retrieval, the model call, the abstention behavior, all the work of the earlier chapters, sits behind that chat box unchanged. This is what it looks like, running:
ImportantCompliance: say that it is AI
A user-facing chat interface must disclose that the user is talking to an AI system; in the EU that is a legal obligation under the AI Act’s transparency rules, and it is good practice everywhere. One line in the interface (“answers are generated by an AI system from your company’s documents; verify before acting”) satisfies the spirit and most of the letter. Appendix D covers the wider obligations.
25.4 What is deployed with it
The application is the easy part. What separates a demo from a product is what is deployed alongside it, and it is exactly the discipline this book has built:
- The eval harness (Chapter 9), wired to run on every change, so you know before you deploy whether a prompt or model update helped or hurt.
- Tracing and cost tracking (Chapter 24), on from day one, so the first production incident is debuggable and the bill is attributable.
- Drift monitoring (Chapter 24) on the incoming questions, so you hear about a shift before users complain.
- Guardrails (Chapter 23), because the chat box is untrusted input, and output validation, because the answer feeds a real decision.
A system deployed with these is observable, measurable, and defensible, while one deployed without them is a liability behind an attractive interface.
flowchart TB
colleague(["colleague"]) --> ui["Chainlit chat UI<br/>chat.py"]
ui --> svc["FastAPI service, app.py:<br/>/ask, /triage, /analyst"]
svc --> ra["rag_answer<br/>grounded answers (Ch 7)"]
svc --> tr["structured triage<br/>(Ch 3)"]
svc --> ag["agent with tools<br/>(Ch 12)"]
ra --> llm["models via OpenRouter"]
tr --> llm
ag --> llm
evh["eval harness (Ch 9)<br/>runs on every change"] -.- svc
obs["tracing and drift<br/>monitoring (Ch 24)"] -.- svc
The first bullet already runs, against the live service: five golden questions with hand-checked answers, including one out-of-scope question that the system must abstain on, posted through the same HTTP path a user’s request would take. This is the gate a deploy must pass.
golden = [
("How many vehicles did Tesla deliver in 2023?", lambda a: "1.8" in a),
("What was Amazon's total net sales in 2023, in billions?", lambda a: "574" in a),
("What business is Hormel Foods in?", lambda a: "food" in a.lower()),
("What does Park Hotels own or operate?", lambda a: "hotel" in a.lower()),
# Out of scope: the corpus has no Apple filing, so the only pass is an abstention.
("What was Apple's iPhone revenue in 2023?",
lambda a: any(w in a.lower() for w in ["cannot", "not in the", "no information"])),
]
passed = sum(check(client.post("/ask", json={"question": q}).json()["answer"])
for q, check in golden)
print(f"gate: {passed}/{len(golden)} pass")gate: 3/5 passFive questions amount to a smoke test, whereas the real gate, the Chapter 9 harness, runs the full eval set. But even this small, it is the difference between “the demo worked when I tried it” and a pass count a deploy script can refuse to deploy on.
TipCost: what one user costs per day
The service’s unit economics are the sum of its parts, all measured in earlier chapters: a grounded answer costs a fraction of a cent (Chapter 7), a triage call less (Chapter 3), and an agent question a few times more because of the loop (Chapter 12). A colleague asking thirty questions a day costs a few cents. The tracing from Chapter 24 is what turns that estimate into an invoice you can attribute per team. On self-hosted hardware the same mix appears as load, since an agent call occupies the hardware for several model calls in sequence, so the traffic split across endpoints sets throughput and capacity the way it sets the invoice here. The expensive failure, which dwarfs the bill, is deploying without the harness and finding out late that the answers drifted.
Those per-call figures turn into a team’s monthly bill with three sliders. These unit costs are rounded versions of the ones measured in Chapters 3, 7, and 12. Swap in your own once your tracing reports them.
25.5 The pilot scorecard
Industry surveys through 2025 and 2026 reported a sobering pattern: a large share of generative-AI pilots never reached production. Although the models themselves were rarely at fault, the pilots lacked everything around the model: evaluation, monitoring, a clear owner, and a credible measure of value. So before you build the service, scope the pilot to graduate. A pilot worth running has, written down before it starts: one well-defined task with a baseline (what is the current cost or accuracy you are trying to beat), a success metric and a threshold (Chapter 9’s discipline, decided in advance), a named owner, and a real cost-per-outcome in Chapter 2’s unit-economics terms (a common mistake is to quote dollars per token). If you cannot fill in that scorecard, the problem is not yet ready for a model, and building the service first will not change that.
WarningDon’t outsource this
No assistant can fill in your pilot’s scorecard. The baseline cost of a human routing a ticket, the accuracy threshold you would actually trust in production, the named owner who answers for the tool, and the true cost-per-outcome are all judgments about your organization that a model has no access to and no standing to make. Draft the table’s structure with a coding tool if you like; the four numbers in the last column are yours to defend before the pilot justifies a single line of service code.
Here is the scorecard as a form, with the middle column filled in for the ticket-triage assistant this book has carried since Chapter 2, and the last column waiting for your pilot.
| scorecard line | the book’s ticket-triage assistant | your pilot |
|---|---|---|
| task and its baseline | triage incoming support tickets into the fifteen categories; today a person routes each one at roughly $0.50 of handling time per ticket | |
| success metric and threshold | category accuracy of at least 0.92 on the Chapter 9 golden set, fixed before the pilot started | |
| owner | the support operations lead, by name | |
| cost per outcome | about $0.0002 per triaged ticket (Chapter 2’s measurement), against the $0.50 baseline |
25.6 Evaluation: is the assembled system ready?
Because no single metric captures readiness, the final evaluation is a checklist of the production-readiness questions you must be able to answer yes to.
| Readiness check | Status |
|---|---|
| Answers correctly on the eval set (Ch 9) | measured |
| Abstains on out-of-scope questions (Ch 7) | measured |
| Cost per answer is known (Ch 2) | measured |
| Calls are traced (Ch 24) | wired in |
| Inputs are monitored for drift (Ch 24) | wired in |
| Untrusted input is guarded (Ch 23) | wired in |
| A human approves irreversible actions (Ch 23) | by design |
Every row of this checklist is a chapter of this book, which is the point: a system you would deploy is the answer function plus the discipline around it, and you have built both. The chat box amounts to the last five lines of code, while the trustworthiness comes from everything built before it.
25.7 What you can do now
You started this book able to call a model. You have now built working versions of the things that matter: a retrieval system over real documents that you proved is better than the alternative, an agent that uses tools and a sense of when not to reach for one, a fine-tuned model and the judgment of whether it was worth it, synthetic data and a hard look at when it helps, pipelines that read audio and images and time series, a knowledge graph and the traversal that answers multi-hop questions, a defense against input crafted to subvert the system, and the instrumentation to detect trouble in production before users do. Some of these you ran at full scale and some at demo scale, but in every case you saw the real mechanism, so no capability remains a black box. Most of all, you can measure each of them, because every capability here came with the question that matters more than how to build it: how do you know it works? This question, asked and answered with a number, is the difference between a demo and a system people rely on, and you are now equipped to build one.
25.8 Choosing your production stack
Part IX wrapped the system in a service, watched it in production, and guarded it. These are the layers that turn a notebook into something a colleague relies on.
| Capability | Tools | Notes | Choose by |
|---|---|---|---|
| Serving and UI | FastAPI or Ray Serve, with Chainlit, Streamlit, Gradio | an API plus a chat front end | who uses it and how |
| Observability | Langfuse, Arize Phoenix, Helicone | tracing of cost, latency, and output | self-host against a hosted dashboard |
| Guardrails and gateway | NeMo Guardrails, Guardrails AI, Llama Guard; LiteLLM or Portkey gateway | screen input, centralize keys and spend | risk surface and scale |
Important
- Deploy the eval harness and tracing with the application from the first day (Chapters 24, 25).
- Keep a human gate on any destructive action the system can take (Chapter 23).
- Let a pilot scorecard decide whether it graduates to production; a common mistake is to let enthusiasm decide (Chapter 25).
Common failure points
- Deploying with no tracing, so drift goes unseen until a user complains (Chapter 24).
- Acting on untrusted input without guardrails, which prompt injection turns against you (Chapter 23).
- Leaving no kill switch or rollback, so a bad deploy has no exit (Chapter 24).
The current open-source and vendor options for these are in the tooling-landscape appendix, dated and fuller; model and provider choice is Appendix E.
25.9 Exercises
25.9.1 Conceptual questions
How much new modeling code did the move from notebook to product require?
- A rewrite of the retrieval layer to handle concurrent requests
- A new, faster model to keep latency acceptable over HTTP
- Almost none; the existing functions were wrapped in an interface
- A rebuilt index, because the notebook’s vector store cannot serve concurrent web requests
According to the chapter, the main reason most generative-AI pilots never reach production is:
- Missing discipline around the model: evaluation, monitoring, ownership, value measurement
- Model quality that looks fine in a demo but collapses once it meets real traffic
- API costs that grow faster than the value the pilot delivers to the business
- Interfaces too rough for business users to adopt without training
What is deployed alongside the application to turn a demo into a product?
- A higher-capacity model and a much larger document corpus
- A service-level agreement, a status page, and an on-call rotation
- Load balancing, autoscaling, and a private container registry
- The eval harness, tracing, drift monitoring, and guardrails
A pilot scorecard worth running specifies, before the pilot starts:
- The model, the framework, and the deployment platform it will use
- The number of expected users, the launch date, and the demo script
- A baselined task, a success metric and threshold, an owner, and a cost-per-outcome
- The GPU budget, the context window, and the negotiated token price
The question that separated a demo from a system throughout the book is:
- How fast does it respond under production load?
- How do you know it works, answered with a number?
- Which model offers the largest context window?
- How many documents can the index hold before it slows down?
In the FastAPI service, typing the request as a Pydantic model means:
- Malformed requests are rejected before they ever reach the model
- The model’s answers are validated against a response schema
- Each request is automatically traced with its tokens, cost, and latency
- The question is embedded and cached before retrieval runs
Why do sources travel with every answer the service returns?
- The trace store requires a source field on every record it stores
- The service bills each answer back to the owner of the source document
- FastAPI requires every response field to be declared in the schema
- So the caller can always check the grounding of the answer
The chapter’s final evaluation is a readiness checklist, whereas earlier chapters reported a metric, because:
- The earlier chapters already exhausted the eval set, leaving nothing new to measure
- Readiness mixes measured results with wired-in safeguards; no one number captures both
- A checklist is cheaper to maintain than a judge-based metric run on every change
- Production systems change too quickly after launch for any single metric to remain valid
25.9.2 Build lab
Take rag_answer and wrap it in the FastAPI app above. Run it locally with uvicorn and call the /ask endpoint with a question. Confirm you get an answer and sources over HTTP. Then add one thing from the readiness checklist that is not yet wired in, and describe what it took.
25.9.3 Evaluate lab
Fill in the pilot scorecard for a real problem at your organization: the task, the baseline, the success metric and threshold, the owner, and the cost-per-outcome. If any row is blank, write one sentence on what you would need to fill it. Decide whether this pilot is ready to build, the same judgment that separates the projects that reach production from the ones that are quietly abandoned.
TipProject ideas
You can now defend a system against prompt injection, validate its output, trace what it does in production, watch for drift, and wrap it in a service. These two projects take something you built earlier to a state where you could put it in front of users.
- Harden and instrument a system you built. Wrap one of your earlier projects in a small service, add output validation and tracing, and then try to break it by feeding it prompt-injection attempts, both direct and hidden inside retrieved documents. Measure how often the guardrails hold. Data to try: a public prompt-injection or jailbreak set, with your own application as the target.
- Run a pilot scorecard. Put a system in front of a small group of real users, sample its traffic, and score a slice of responses each week. Watch for drift in quality and cost. Decide in advance what numbers would justify a wider rollout. Data to try: your own application’s traffic.
Part X works three projects end to end, from raw data to a measured, deployed system. You now have what you need to carry any of these, or a system of your own, the whole way there.
NoteThe end, and the beginning
This completes the book. You now have the toolkit and, more importantly, the discipline to use it well. The appendices cover setup, benchmarking, cost modeling, compliance, model selection, and working alongside AI coding tools; reach for them as you build, because everything beyond this point is practice.