12 Agents and MCP

Letting the model drive, and connecting it to the world

In a workflow, we wrote the control flow. An agent hands that decision to the model: we give it a set of tools and a goal, and it decides at each step whether to call a tool, which one, and when it has enough to answer. This flexibility is powerful and easy to overspend on, so this chapter does three things. It builds a small agent and watches it choose and chain tools. It shows why agents need a model trained for reliable tool use, and measures what happens without one. And it introduces the Model Context Protocol (MCP), the standard that lets an agent reach tools and services beyond the ones it was built with, the same standard your coding assistant uses to reach external systems beyond its built-in tools. By the end you could write a tool for that assistant yourself.

In the running system

This chapter builds our desk’s agent path, the one that handles a question with steps, such as pulling a figure from the filings and comparing it to last year. It arrives in Chapter 25 as /analyst, alongside triage and document answering.

Setup for this chapter

Run in the gaba-core environment with an OPENROUTER_API_KEY. This chapter introduces gaba.agent. The agent departs from the book’s default model and uses MODEL_AGENT, a model chosen for reliable tool use, for reasons the chapter explains.

from dotenv import load_dotenv
load_dotenv()

True

12.1 The agent loop

An agent is a loop: call the model with the question and a list of tools; if it replies with a tool call, run that tool, hand the result back, and call the model again; if it replies with text and no tool call, it is done. This loop is the whole mechanism, and gaba.agent.run_agent is about fifty lines that implement exactly it. What makes it feel intelligent is the model deciding, at each turn, what to do next; the loop itself is plumbing.

flowchart TB
    q([question + tool list]) --> step["call the model"]
    step --> decide{"did it reply with<br/>a tool call?"}
    decide -- "yes" --> run["run the tool"]
    run --> feed["append the result<br/>to the conversation"]
    feed --> step
    decide -- "no: plain text" --> done([final answer])

Figure 12.1: The agent loop. The diamond is the only decision point, and the model makes that decision. Everything an agent does, however long the trace, is this loop repeating.

12.2 Giving an agent tools

A tool is two things: a Python function that does something, and a schema that describes it to the model so the model knows when and how to call it. We give our agent three: a financial lookup, a calculator, and a search over the report corpus from earlier chapters.

from gaba.rag import retrieve

# A small store of figures the lookup tool serves. In a real system this would
# be a database query; the agent does not know or care which.
FACTS = {
    "TSLA": {"deliveries": 1_808_581, "revenue": 96_770_000_000},
    "AMZN": {"net_sales": 574_800_000_000, "aws_revenue": 90_800_000_000},
    "HRL": {"net_sales": 12_100_000_000},
    "BKH": {"revenue": 2_560_000_000},
}
ALIASES = {"tesla": "TSLA", "amazon": "AMZN", "hormel": "HRL", "black hills": "BKH"}

def lookup_financial(company: str, metric: str):
    """Look up a stored figure by company name or ticker and metric name."""
    ticker = ALIASES.get(company.lower().strip(), company.upper())
    return FACTS.get(ticker, {}).get(metric, "not found")

def calculator(expression: str):
    """Evaluate an arithmetic expression. (A real system would use a safe parser.)"""
    return eval(expression, {"__builtins__": {}}, {})

def search_reports(query: str):
    """Search the annual-report corpus for a passage."""
    return " | ".join(f"[{h['ticker']}] {h['text'][:100]}" for h in retrieve(query, n=2))

tools = {"lookup_financial": lookup_financial, "calculator": calculator,
         "search_reports": search_reports}

The schemas tell the model what each tool expects. This is the same JSON-schema idea as Chapter 3’s structured outputs, pointed the other way: there we described the output we wanted; here we describe the inputs each tool needs.

schemas = [
    {"type": "function", "function": {
        "name": "lookup_financial",
        "description": "Look up a stored financial figure by company (name or ticker) "
                       "and metric (deliveries, revenue, net_sales, aws_revenue).",
        "parameters": {"type": "object", "properties": {
            "company": {"type": "string"}, "metric": {"type": "string"}},
            "required": ["company", "metric"]}}},
    {"type": "function", "function": {
        "name": "calculator", "description": "Evaluate an arithmetic expression.",
        "parameters": {"type": "object",
            "properties": {"expression": {"type": "string"}}, "required": ["expression"]}}},
    {"type": "function", "function": {
        "name": "search_reports", "description": "Search company annual reports for a passage.",
        "parameters": {"type": "object",
            "properties": {"query": {"type": "string"}}, "required": ["query"]}}},
]

Do not put that calculator into production

The calculator tool runs Python’s eval with an emptied builtins dict. This is fine for a demonstration and unsafe in production: emptying __builtins__ is not a real sandbox, and known escapes exist. A tool that runs eval on text the model controls, especially one you expose over MCP for any client to call (as in the server below), is a remote-code-execution risk. For anything real, replace eval with a dedicated arithmetic parser.

Now run the agent on a question that no single tool answers: it needs to look up a figure and then do arithmetic on it.

from gaba.agent import run_agent

result = run_agent(
    "What is 5 percent of Tesla's 2023 vehicle deliveries?",
    tools=tools,
    schemas=schemas,
    system="You are a financial analyst assistant. Use the tools to look up figures and do arithmetic.",
)

print("tool calls the agent chose:")
for step in result.steps:
    print(f"  {step['tool']}({step['args']}) -> {step['result']}")
print("\nfinal answer:", result.answer)

tool calls the agent chose:
  lookup_financial({'company': 'Tesla', 'metric': 'deliveries'}) -> 1808581
  calculator({'expression': '1808581 * 0.05'}) -> 90429.05

final answer: 5 percent of Tesla's 2023 vehicle deliveries is approximately 90,429 vehicles.

The agent decided, with no prompting about which tools to use, to look up the delivery figure and then pass it to the calculator; the model chose both of those steps on its own, without our writing either one. This run-time choice of steps is the difference from a workflow, and it is what justifies the added expense of an agent when the steps cannot be known in advance.

12.3 A tool that runs SQL

Real agents, which rarely work from hand-curated facts, sit in front of databases, which they query in SQL (Structured Query Language, which Chapter 14 covers in full). We give ours Chapter 14’s warehouse in miniature, eight companies and their 2023 financials, exposed as one more tool. Two details carry the lesson. The connection is read-only (PRAGMA query_only = ON), Chapter 14’s safety rule, because this query will be written by a model. And the tool’s description tells the agent what the database can and cannot answer, because with two overlapping data tools (lookup_financial and this one), the description is what routes the call.

import sqlite3

db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE financials (ticker TEXT, name TEXT, year INT,
                         revenue_billions REAL, net_income_billions REAL);
INSERT INTO financials VALUES
  ('AMZN','Amazon',2023,574.8,30.4),   ('TSLA','Tesla',2023,96.8,15.0),
  ('BAC','Bank of America',2023,98.6,26.5), ('HRL','Hormel Foods',2023,12.1,1.0),
  ('BKH','Black Hills',2023,2.6,0.25), ('AMBC','Ambac',2023,1.2,0.04),
  ('ATSG','Air Transport Services',2023,2.0,0.06), ('PK','Park Hotels',2023,2.6,0.10);
""")
db.execute("PRAGMA query_only = ON")

def query_database(sql: str):
    """Run a read-only SQL query against the financials table."""
    try:
        cur = db.execute(sql)
        cols = [d[0] for d in cur.description]
        return [dict(zip(cols, row)) for row in cur.fetchmany(20)]
    except Exception as exc:
        return f"SQL error: {exc}"

tools["query_database"] = query_database
schemas.append({"type": "function", "function": {
    "name": "query_database",
    "description": "Run a read-only SQL query against financials(ticker, name, "
                   "year, revenue_billions, net_income_billions). 2023 data for "
                   "eight companies. Use for rankings, ratios, and aggregates.",
    "parameters": {"type": "object",
        "properties": {"sql": {"type": "string"}}, "required": ["sql"]}}})

The error path does quiet work here: a failed query, which would ordinarily raise, returns the error message to the agent, so the model can read it and rewrite the SQL, the same self-correction loop Chapter 14 built deliberately, happening here on the agent’s own initiative.

Now we ask a question that no single tool answers, one that crosses from structured data into documents:

result = run_agent(
    "Which of the eight companies had the highest net margin in 2023, "
    "and what does its annual report say about its strategy?",
    tools=tools, schemas=schemas,
    system="You are a financial analyst assistant. Use the tools.",
)
for step in result.steps:
    print(f"  {step['tool']}({str(step['args'])[:60]}) -> {str(step['result'])[:80]}")
print("\nfinal answer:", result.answer[:400])

  query_database({'sql': 'SELECT name, revenue_billions, net_income_billions,) -> [{'name': 'Bank of America', 'revenue_billions': 98.6, 'net_income_billions': 26
  search_reports({'query': 'strategy Bank of America 2023'}) -> [BAC] Bank of America also knows that in addition to high-tech, many clients wil

final answer: In 2023, Bank of America had the highest net margin among the eight companies, with a net margin of approximately 26.9%. 

Regarding its strategy, the annual report for Bank of America mentions a focus on high-tech solutions combined with personalized client service. The management team emphasizes a client-centered approach alongside technological advancements. Would you like a more detailed summa

The agent wrote a SQL query with a computed ratio, read the result, and carried the winning company into a document search, structured and unstructured retrieval composed in one loop, with no step of that plan written by us. This is the structure we see in most production agents today: a handful of data tools with accurate descriptions, a model that routes between them, and read-only access everywhere a model writes the query.

We claimed the description is what routes the call, so we measure it. We register the same SQL tool a second way, with its description stripped down to “Run a SQL query.”, and measure the effect.

Metric: correct-first-tool rate, the fraction of questions where the agent’s first call goes to query_database.
Test set: four questions about rankings and aggregates across the eight companies, exactly what the database is for.
Baseline: the rich description the tool was built with.

import copy
import pandas as pd

schemas_degraded = copy.deepcopy(schemas)
for s in schemas_degraded:
    if s["function"]["name"] == "query_database":
        s["function"]["description"] = "Run a SQL query."

routing_questions = [
    "Which of the eight companies had the highest net margin in 2023?",
    "Rank the eight companies by 2023 net income.",
    "What was the average revenue across all eight companies in 2023?",
    "How many of the companies earned more than 10 billion dollars of net income in 2023?",
]

def first_tool_rate(schema_set):
    hits = 0
    for q in routing_questions:
        r = run_agent(q, tools, schema_set,
                      system="You are a financial analyst assistant. Use the tools.")
        hits += bool(r.steps) and r.steps[0]["tool"] == "query_database"
    return hits

n_q = len(routing_questions)
rich_hits = first_tool_rate(schemas)
degraded_hits = first_tool_rate(schemas_degraded)

pd.DataFrame({
    "tool description": ["rich: schema, coverage, and when to use it", "degraded: \"Run a SQL query.\""],
    "correct first tool": [f"{rich_hits}/{n_q}", f"{degraded_hits}/{n_q}"],
})

	tool description	correct first tool
0	rich: schema, coverage, and when to use it	4/4
1	degraded: "Run a SQL query."	1/4

The rich description tells the model what the database contains and when to reach for it; the degraded one leaves it to guess between four tools, and the gap between the two rows measures what that guessing loses in routing accuracy. The practical lesson is that a tool’s description, although it reads like documentation, works as a prompt: it is the only thing the model has to route on, and writing it is part of building the tool, the same way Chapter 11’s orchestrator needed to be told what its workers could answer.

12.4 Agents need a model trained for reliable tool use

We built a catch into the setup. The agent runs on MODEL_AGENT, the agent model, one trained and tested for reliable tool calling, although every earlier chapter reached for the book’s default lightweight model. The distinction is a specific skill, one separate from raw intelligence: emitting a well-formed tool call every time one is needed, passing the right arguments, and folding each result into the next decision, over and over without dropping a step. A model without that training is an unreliable agent however well it answers single prompts: it forgets to call tools, calls them with the wrong arguments, or answers from memory. This is the routing lesson from Chapter 2 at higher stakes: send simple classification to the default model, but put a tool-reliable model behind an agent loop. The price difference between the two is modest, so the lesson concerns reliability; what actually grows an agent’s bill is the loop, several calls per question with a context that lengthens at each step. The evaluation at the end of this chapter shows the reliability gap directly: the same agent that solves nearly every task on the agent model solves almost none on the default one.

Table 12.1: What reliable tool use requires, and where each model stands. The last two rows are the trap: on single prompts and on price the default model looks like the obvious choice, and the loop is where it falls apart.

what the loop demands	tool-trained agent model	default lightweight model
emits a well-formed tool call whenever one is needed	reliably	often skips the call and answers from memory
passes the right arguments	reliably	drops or garbles arguments
folds each result into the next decision	step after step	loses the thread mid-loop
single-prompt quality (classify, extract, draft)	strong	strong, which is why it fooled us in Chapter 2
price per call	a modest premium	cheapest

12.5 Connecting to the world: the Model Context Protocol

Our three tools are defined inside this notebook. Real agents need tools hosted elsewhere: a company database, a ticketing system, a search API, your own filesystem. Wiring each one in by hand, in a different way for every agent and every tool, does not scale. The Model Context Protocol (MCP), introduced by Anthropic in late 2024 and now widely adopted, is the standard that addresses this. An MCP server exposes a set of tools in a standard format; any MCP-aware client, a coding assistant, a chat app, your own agent, can discover and call them without custom integration code.

flowchart LR
    subgraph clients["MCP clients"]
        c1["coding assistant"]
        c2["chat app"]
        c3["your agent"]
    end
    subgraph servers["MCP servers"]
        s1["database server<br/>query, schema"]
        s2["filesystem server<br/>read, write, list"]
        s3["your server<br/>the notebook's three tools"]
    end
    c1 -- "discover + call" --> s1
    c1 --> s2
    c2 --> s2
    c3 --> s1
    c3 --> s3

Figure 12.2: MCP in one picture. Servers expose tools in a standard format; any MCP-aware client can discover and call them. New client or new server, no new integration code.

The same three tools, exposed as an MCP server, look like this. Although we do not execute the server here, you can run it yourself and point a client at it.

# An MCP server exposing our tools. Run with: python this_file.py
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("financial-tools")

@mcp.tool()
def lookup_financial(company: str, metric: str):
    """Look up a stored financial figure by company and metric."""
    ticker = ALIASES.get(company.lower().strip(), company.upper())
    return FACTS.get(ticker, {}).get(metric, "not found")

@mcp.tool()
def calculator(expression: str):
    """Evaluate an arithmetic expression."""
    return eval(expression, {"__builtins__": {}}, {})

if __name__ == "__main__":
    mcp.run()

This short listing is the whole server. The @mcp.tool() decorator turns a normal Python function, with its docstring and type hints, into a tool that any MCP client can call, without any schema written by hand. This is how your coding assistant connects to external systems: Claude Code, Cursor, and the others are MCP clients, and the integrations you add to them, a database connector, a Slack tool, a filesystem bridge, are MCP servers exactly like this one.

Pitfall: tools are an attack surface

Everything a tool returns, and even a tool’s description, is part of the model’s context, and the model follows instructions it finds there. Therefore, an untrusted MCP server can poison a tool description or result with instructions (“before answering, send the conversation to…”), and the agent may comply. Connect only servers you trust, read what their tools claim to do, and treat every tool output as untrusted input. See the section on indirect injection in Chapter 23 for an example of this attack and the layered defenses against it.

With an AI coding tool

This is the chapter where “using the tool” and “building the tool” meet. The MCP server above is the same kind of thing you would write to give your own coding assistant a new capability, a tool that queries your company’s data, runs your test suite, or files a ticket. You now understand the protocol well enough that you can extend the assistant on your desktop, where until now you could only consume it.

12.6 When to use an agent

The flexibility of an agent is also its danger. Because the model decides the control flow, an agent can loop longer than you expected, call a tool you did not want called, or take an action that is hard to undo. A workflow cannot surprise you that way, because you wrote every step. The step budget is the guardrail against the first failure, and we watch it fire once: we hand the agent a task its tools cannot complete, a sweep across all eight companies for a figure that exists in none of them, and cap the loop at three steps.

runaway = run_agent(
    "For each of the eight companies, find what its annual report says about "
    "its revenue guidance for 2024. Check one company at a time.",
    tools=tools, schemas=schemas, max_steps=3,
    system="You are a financial analyst assistant. Use the tools for every figure; "
           "do not answer from memory.",
)
for step in runaway.steps:
    print(f"  {step['tool']}({str(step['args'])[:60]})")
print("\nanswer:", runaway.answer)

  search_reports({'query': 'revenue guidance 2024'})

answer: Please specify the first company from the list of eight companies you want me to check for its 2024 revenue guidance in the annual report.

The trace shows an agent doing exactly what it was told, searching company after company for something the corpus does not contain, and the budget cutting it off mid-sweep with the sentinel answer before it could spend eight more calls confirming the absence. This sentinel, which can read as a failure, is the guardrail working: the budget converts what would have been an unbounded bill into a bounded, visible non-answer. In production the sentinel is also a signal to log and alert on, because a rising rate of step-budget exits means users are asking the agent for things its tools cannot do.

So the rule holds: use the simplest thing that works. Reach for a single call when the task is single, a workflow when the steps are known, and an agent only when the steps cannot be known in advance, and even then, with a step budget, a careful set of tools, and a human in the loop for anything irreversible. We return to those guardrails in Part IX.

The agent frameworks you will meet in practice, the OpenAI Agents SDK, LangGraph, the Claude Agent SDK, are wrappers around exactly the loop this chapter built: they add conveniences like tracing, retries, and handoffs between agents, but the call-the-model, run-the-tool, feed-the-result cycle underneath is the one in run_agent, which is why building it raw once is worth the trouble. Two directions sit beyond this book’s scope. Computer-use and browser agents replace the fixed tool list with a screen, keyboard, and mouse, the same loop with a much riskier action space. Multi-agent systems put several agents in conversation with each other, dividing a task the way the orchestrator pattern divided one in Chapter 11, with coordination problems to match.

12.7 Evaluation: does the agent get the answer right?

We judge an agent on whether it completes the task, since a loop that ran to completion proves nothing about the answer. We give it multi-step financial questions whose answers we can compute ourselves, and check the agent’s answer against them. The test set covers all four tools: the first three tasks need a lookup and a calculation, the next three need the SQL tool (we verified each answer by hand against the table: Bank of America’s 26.5 on 98.6 is the highest net margin at about 26.9 percent, total revenue sums to 790.7 billion, and Ambac’s 1.2 billion is the smallest), and the last needs the report search, scored by whether a phrase from the actual filing appears in the answer.

Metric: task success, the fraction of questions answered correctly, with the spend and the average number of tool steps reported alongside.
Test set: seven questions spanning the lookup, calculator, SQL, and report-search tools.
Baseline: the same agent on the book’s default lightweight model, to test the claim that the agent loop needs a model trained for tool use.

from gaba.llm import MODEL_AGENT, MODEL_DEFAULT

tasks = [
    ("What is 5 percent of Tesla's 2023 vehicle deliveries?", ["90,429", "90429"]),
    ("What is Amazon's net sales plus AWS revenue in 2023, in billions of dollars?",
     ["665.6", "665,600", "665600"]),
    ("Amazon's AWS revenue is what percentage of its net sales in 2023?",
     ["15.7", "15.8", "16"]),
    ("Which of the eight companies had the highest net margin in 2023?",
     ["bank of america", "bac"]),
    ("What was the total 2023 revenue of all eight companies combined, in billions?",
     ["790.7", "790,700", "790700"]),
    ("Which company had the lowest revenue in 2023?", ["ambac", "ambc"]),
    ("According to its annual report, what is Tesla's stated mission?",
     ["sustainable energy"]),
]

def evaluate_agent(model: str) -> tuple[int, float, float]:
    """Return tasks solved, total cost, and average tool steps per task."""
    solved, cost, steps = 0, 0.0, 0
    for question, accepted in tasks:
        result = run_agent(question, tools, schemas, model=model,
                           system="You are a financial analyst. Use the tools.")
        answer = result.answer.replace("$", "").lower()
        solved += any(a in answer for a in accepted)
        cost += result.cost_usd
        steps += len(result.steps)
    return solved, cost, steps / len(tasks)

agent_solved, agent_cost, agent_steps = evaluate_agent(MODEL_AGENT)
default_solved, default_cost, default_steps = evaluate_agent(MODEL_DEFAULT)

import pandas as pd
pd.DataFrame({
    "model": [f"{MODEL_AGENT} (agent model)", f"{MODEL_DEFAULT} (default)"],
    "tasks solved": [f"{agent_solved}/{len(tasks)}", f"{default_solved}/{len(tasks)}"],
    "avg steps": [f"{agent_steps:.1f}", f"{default_steps:.1f}"],
    "cost $": [f"{agent_cost:.5f}", f"{default_cost:.5f}"],
})

	model	tasks solved	avg steps	cost $
0	openai/gpt-4.1-mini (agent model)	7/7	1.7	0.00242
1	google/gemini-2.5-flash-lite (default)	1/7	0.4	0.00055

This is the “agents need a model trained for reliable tool use” claim made concrete, and the gap is not subtle. The agent model solves most or all of the tasks by composing its tools, getting figures it could not reliably recall and arithmetic it could not reliably do (where it drops one, the trace shows a slip from which it recovers). The default model, the very one that excelled at single-prompt classification back in Chapter 2, solves few or none, and its failures trace to the loop itself, since the arithmetic is well within its reach: it forgets to call the tools, calls them with the wrong arguments, or tries to answer from memory. The avg steps column is the loop made visible: an agent doing real multi-step work averages a couple of tool calls per task, and a number far from that, in either direction, is a symptom worth investigating in the trace. The cost column makes the same point from the other side: the whole comparison cost well under a cent, so the gap, which price cannot explain at these amounts, comes from whether the model was trained to drive the loop at all. When an agent fails, the trace, the list of steps, tells you which of those went wrong, which is why an agent’s evaluation covers the path it took as well as the final answer; the Evaluate lab below puts that diagnosis in your hands.

Don’t outsource this

If an agent misses a task, an assistant will readily propose another tool, a longer system prompt, or a retry wrapper, and one of those fixes might even work. But deciding what result.steps actually shows went wrong, whether a skipped call, a garbled argument, or an answer pulled from memory where a lookup was needed, is not a judgment you can hand back to the assistant that just watched the agent fail. Read the trace yourself before accepting any fix, because the fix that satisfies the assistant is not automatically the fix that addresses the failure you diagnosed.

Cost: agents are the expensive option

An agent’s cost is driven by its loop. A few thousand tokens through the agent model still costs a fraction of a cent, a modest premium over the default; what grows the bill is that an agent makes several calls per question, each carrying the entire conversation so far, so a ten-step run pays for ten increasingly long contexts. On hardware you host, the same growth appears as latency and capacity: each call’s longer context takes longer to process and occupies the GPU for more of the run, so one agent question displaces many single-call questions. This is fine when the task needs it and ruinous when it does not, which is the whole reason the rule is to escalate to an agent only after the simpler designs have been ruled out.

The growth of “increasingly long” is quadratic, although the phrase suggests something linear, and the arithmetic shows why: if each step adds roughly the same number of tokens, call $k$ pays for about $k$ steps’ worth of context, and a run of $N$ steps pays for about $N(N+1)/2$ steps’ worth in total.

viewof context_growth = {
  let steps = 6, perStep = 300;
  const container = document.createElement("div");
  container.style.cssText = "font-family: inherit; color: var(--bs-body-color, #1f2328); border: 1px solid var(--bs-border-color, #d0d7de); border-radius: 6px; padding: 12px; max-width: 640px;";

  function render() {
    container.innerHTML = "";
    const mkSlider = (labelText, min, max, step, value, onInput) => {
      const row = document.createElement("div");
      row.style.cssText = "display: flex; gap: 10px; align-items: center; margin-bottom: 6px; font-size: 13px;";
      const lab = document.createElement("span");
      lab.style.cssText = "color: var(--bs-secondary-color, #57606a); min-width: 210px;";
      lab.textContent = `${labelText}: ${value.toLocaleString()}`;
      const s = document.createElement("input");
      s.type = "range"; s.min = min; s.max = max; s.step = step; s.value = value;
      s.style.flex = "1";
      s.oninput = () => { onInput(+s.value); render(); };
      row.append(lab, s);
      return row;
    };
    container.appendChild(mkSlider("steps in the loop", 1, 12, 1, steps, v => { steps = v; }));
    container.appendChild(mkSlider("tokens added per step", 100, 1000, 50, perStep, v => { perStep = v; }));

    const maxLen = steps * perStep;
    const bars = document.createElement("div");
    bars.style.cssText = "margin: 10px 0 6px;";
    for (let k = 1; k <= steps; k++) {
      const row = document.createElement("div");
      row.style.cssText = "display: flex; gap: 8px; align-items: center; margin-bottom: 3px; font-size: 12px;";
      const lab = document.createElement("span");
      lab.style.cssText = "min-width: 52px; color: var(--bs-secondary-color, #57606a);";
      lab.textContent = `call ${k}`;
      const track = document.createElement("div");
      track.style.cssText = "flex: 1; height: 13px; background: rgba(128,128,128,0.18); border-radius: 3px; overflow: hidden;";
      const fill = document.createElement("div");
      fill.style.cssText = `width: ${(100 * k * perStep / maxLen).toFixed(1)}%; height: 100%; background: #0969da;`;
      track.appendChild(fill);
      const val = document.createElement("span");
      val.style.cssText = "min-width: 88px; text-align: right;";
      val.textContent = `${(k * perStep).toLocaleString()} tokens`;
      row.append(lab, track, val);
      bars.appendChild(row);
    }
    container.appendChild(bars);

    const total = perStep * steps * (steps + 1) / 2;
    const readout = document.createElement("div");
    readout.style.cssText = "font-size: 13px; margin-top: 8px;";
    readout.innerHTML = `Total prompt tokens across the run: <b>${total.toLocaleString()}</b>, ` +
      `which is <b>${(steps * (steps + 1) / 2).toLocaleString()}x</b> what a single call over the same material would pay.`;
    container.appendChild(readout);
  }

  render();
  return container;
}

Figure 12.3: What an agent loop pays for. Each bar is one call in the loop, and its length is the context that call carries: everything every earlier step added. Drag the sliders and watch the total grow roughly with the square of the step count, which is why a long agent run costs far more than its step count suggests.

12.8 Choosing your workflow and agent stack

In Part IV we built workflows by hand and an agent loop, then connected tools through MCP. Frameworks exist for each, and most projects need fewer of them than it seems.

Capability	Lightweight	Framework	Choose by
Orchestration	plain Python	LangGraph, LlamaIndex Workflows	how complex the control flow really is
Agent runtime	the book’s fifty-line loop	OpenAI Agents SDK, Pydantic AI, CrewAI	whether you need the loop managed for you
Tools	direct function calls	MCP via FastMCP and reference servers	reuse across clients

Important

Prefer a workflow you control to an agent that decides for itself; it is cheaper and easier to debug (Chapter 11).
Give an agent a model trained for tool use, because a general model chains tools unreliably (Chapter 12).
Cap the step count, which stops a confused agent that would otherwise loop forever.

Common failure points

Reaching for an agent when a workflow would do (Chapter 11).
Giving an agent a destructive tool with no human gate (Chapter 23).
Adopting a heavy framework for a flow that three functions would express.

The current open-source and vendor options for these are in the tooling-landscape appendix, dated and fuller; model and provider choice is Appendix E.

12.9 Exercises

12.9.1 Conceptual questions

What distinguishes an agent from a workflow?
1. an agent can call external tools, which a workflow has no way to do
2. the model picks each next step at run time, while a workflow follows steps we wrote
3. an agent runs all of its steps in parallel while a workflow runs them in order
4. an agent needs no system prompt because its tools define the behavior
In the chapter’s evaluation, the agent model solved most of the tasks and the default model solved almost none. The default model failed mainly by:
1. exhausting its context window before it could produce a final answer
2. picking the calculator on questions that the report search should have handled
3. timing out on the network during its longer chains of tool calls
4. breaking down on the loop itself: skipping tools, passing wrong arguments, or answering from memory
What problem does the Model Context Protocol solve?
1. a standard format for tools, so any client can discover and call them with no custom glue
2. a shared memory format, so a model’s conversation history can carry across different applications
3. a verification layer that checks a tool’s output before the model sees it
4. a routing layer that sends each question to the cheapest capable model
When an agent misbehaves, the most useful thing to inspect is:
1. the final answer, since it reflects everything the agent did
2. the system prompt, since agents fail only when instructions are unclear
3. the trace of tool calls and results, which shows where the path broke
4. the temperature, since randomness is the usual cause of a bad loop
The right time to escalate from a workflow to an agent is when:
1. the steps needed to finish the task cannot be predicted in advance
2. the task requires more than one model call to get from question to answer
3. you want the cheapest way to answer a multi-step question
4. the task needs structured output that a workflow cannot enforce
The evaluation runs the same agent on the default lightweight model as its baseline. What makes that the right baseline?
1. the default model is the standard baseline used in published agent benchmarks
2. it proves the test questions are easy enough for any model to answer
3. running both models cuts the cost of building the evaluation in half
4. it isolates the model as the only difference, directly testing the claim that the loop needs a model trained for tool use
The chapter warns against deploying the eval-based calculator, especially over MCP, because:
1. eval is far too slow for production traffic at any real volume
2. it runs model-controlled text as code, and emptied builtins is not a real sandbox
3. MCP clients have no safe way to pass string arguments to a tool
4. arithmetic belongs in the model itself, so the tool adds attack surface for nothing
In the MCP server example, what work does the @mcp.tool() decorator remove?
1. writing the Python function body that does the actual work
2. giving the function a docstring or type hints of any kind
3. hand-writing the JSON schema; the type hints and docstring supply it
4. running a server process at all, since decorated tools are called directly in process

12.9.2 Build lab

Add a fourth tool to the agent, for example one that returns today’s date, or one that converts currencies with a fixed rate, and write a question that requires it together with the existing tools. Run the agent and read the trace to confirm it used the new tool. You may use an assistant to draft the tool and its schema; check the schema matches the function’s arguments yourself.

12.9.3 Evaluate lab

The evaluation showed the default model failing, but the table does not say why. Re-run one failing task on MODEL_DEFAULT and print result.steps, the trace of tool calls it made. Diagnose the failure mode from the trace: did it skip the lookup, call a tool with the wrong argument, or try the arithmetic in its head? Report which step broke, and write one sentence on why the trace is what you actually debug an agent with, given that the score only tells you a task was missed.

Project ideas

You can now build a fixed workflow from the five patterns, escalate to an agent when the task needs it, and give either one tools through MCP. These two projects show you the difference between the two.

A workflow that triages and drafts. Chain the patterns into a pipeline that classifies an incoming request, retrieves what it needs, drafts a reply, and checks its own draft before sending it back. Then build an agent that decides its own steps for the same task, and note where each design is the better fit. Data to try: a support-ticket set, a public email dataset, or your own inbox export.
An analyst agent over a database. Give an agent a SQL tool and a calculation tool, then pose questions that take several steps: pull a slice, compute a metric, compare it across segments, and explain the result. Cap its steps and log every tool call so you can audit its reasoning, then compare it against a fixed workflow that hard-codes the same steps and see which one is worth the extra latency. Data to try: a public relational database such as Chinook, Northwind, or the NYC taxi trips, loaded into SQLite.

Part V turns from answering questions to analyzing text and tables at scale.

Where we go next

That closes Part IV. We can build systems that read, retrieve, reason, and act. In Part V we step back to the data itself: the embeddings we have been using for search hold structure we can mine directly. In Chapter 13 we cluster them to discover the themes in a pile of text, no labels required.

--- title: "Agents and MCP" subtitle: "Letting the model drive, and connecting it to the world" jupyter: gaba-core --- In a workflow, we wrote the control flow. An **agent** hands that decision to the model: we give it a set of tools and a goal, and it decides at each step whether to call a tool, which one, and when it has enough to answer. This flexibility is powerful and easy to overspend on, so this chapter does three things. It builds a small agent and watches it choose and chain tools. It shows why agents need a model trained for reliable tool use, and measures what happens without one. And it introduces the Model Context Protocol (MCP), the standard that lets an agent reach tools and services beyond the ones it was built with, the same standard your coding assistant uses to reach external systems beyond its built-in tools. By the end you could write a tool for that assistant yourself. ::: {.callout-note title="In the running system"} This chapter builds our desk's agent path, the one that handles a question with steps, such as pulling a figure from the filings and comparing it to last year. It arrives in Chapter 25 as `/analyst`, alongside triage and document answering. ::: ::: {.callout-note title="Setup for this chapter"} Run in the `gaba-core` environment with an `OPENROUTER_API_KEY`. This chapter introduces `gaba.agent`. The agent departs from the book's default model and uses `MODEL_AGENT`, a model chosen for reliable tool use, for reasons the chapter explains. ::: ```{python} from dotenv import load_dotenv load_dotenv() ``` ## The agent loop An agent is a loop: call the model with the question and a list of tools; if it replies with a tool call, run that tool, hand the result back, and call the model again; if it replies with text and no tool call, it is done. This loop is the whole mechanism, and `gaba.agent.run_agent` is about fifty lines that implement exactly it. What makes it feel intelligent is the model deciding, at each turn, what to do next; the loop itself is plumbing. ```{mermaid} %%| echo: false %%| label: fig-agent-loop %%| fig-cap: "The agent loop. The diamond is the only decision point, and the model makes that decision. Everything an agent does, however long the trace, is this loop repeating." flowchart TB q([question + tool list]) --> step["call the model"] step --> decide{"did it reply with a tool call?"} decide -- "yes" --> run["run the tool"] run --> feed["append the result to the conversation"] feed --> step decide -- "no: plain text" --> done([final answer]) ``` ## Giving an agent tools A tool is two things: a Python function that does something, and a schema that describes it to the model so the model knows when and how to call it. We give our agent three: a financial lookup, a calculator, and a search over the report corpus from earlier chapters. ```{python} from gaba.rag import retrieve # A small store of figures the lookup tool serves. In a real system this would # be a database query; the agent does not know or care which. FACTS = { "TSLA": {"deliveries": 1_808_581, "revenue": 96_770_000_000}, "AMZN": {"net_sales": 574_800_000_000, "aws_revenue": 90_800_000_000}, "HRL": {"net_sales": 12_100_000_000}, "BKH": {"revenue": 2_560_000_000}, } ALIASES = {"tesla": "TSLA", "amazon": "AMZN", "hormel": "HRL", "black hills": "BKH"} def lookup_financial(company: str, metric: str): """Look up a stored figure by company name or ticker and metric name.""" ticker = ALIASES.get(company.lower().strip(), company.upper()) return FACTS.get(ticker, {}).get(metric, "not found") def calculator(expression: str): """Evaluate an arithmetic expression. (A real system would use a safe parser.)""" return eval(expression, {"__builtins__": {}}, {}) def search_reports(query: str): """Search the annual-report corpus for a passage.""" return " | ".join(f"[{h['ticker']}] {h['text'][:100]}" for h in retrieve(query, n=2)) tools = {"lookup_financial": lookup_financial, "calculator": calculator, "search_reports": search_reports} ``` The schemas tell the model what each tool expects. This is the same JSON-schema idea as Chapter 3's structured outputs, pointed the other way: there we described the output we wanted; here we describe the inputs each tool needs. ```{python} schemas = [ {"type": "function", "function": { "name": "lookup_financial", "description": "Look up a stored financial figure by company (name or ticker) " "and metric (deliveries, revenue, net_sales, aws_revenue).", "parameters": {"type": "object", "properties": { "company": {"type": "string"}, "metric": {"type": "string"}}, "required": ["company", "metric"]}}}, {"type": "function", "function": { "name": "calculator", "description": "Evaluate an arithmetic expression.", "parameters": {"type": "object", "properties": {"expression": {"type": "string"}}, "required": ["expression"]}}}, {"type": "function", "function": { "name": "search_reports", "description": "Search company annual reports for a passage.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}, ] ``` ::: {.callout-warning title="Do not put that calculator into production"} The `calculator` tool runs Python's `eval` with an emptied builtins dict. This is fine for a demonstration and unsafe in production: emptying `__builtins__` is not a real sandbox, and known escapes exist. A tool that runs `eval` on text the model controls, especially one you expose over MCP for any client to call (as in the server below), is a remote-code-execution risk. For anything real, replace `eval` with a dedicated arithmetic parser. ::: Now run the agent on a question that no single tool answers: it needs to look up a figure and then do arithmetic on it. ```{python} from gaba.agent import run_agent result = run_agent( "What is 5 percent of Tesla's 2023 vehicle deliveries?", tools=tools, schemas=schemas, system="You are a financial analyst assistant. Use the tools to look up figures and do arithmetic.", ) print("tool calls the agent chose:") for step in result.steps: print(f" {step['tool']}({step['args']}) -> {step['result']}") print("\nfinal answer:", result.answer) ``` The agent decided, with no prompting about which tools to use, to look up the delivery figure and then pass it to the calculator; the model chose both of those steps on its own, without our writing either one. This run-time choice of steps is the difference from a workflow, and it is what justifies the added expense of an agent when the steps cannot be known in advance. ## A tool that runs SQL Real agents, which rarely work from hand-curated facts, sit in front of databases, which they query in SQL (Structured Query Language, which Chapter 14 covers in full). We give ours Chapter 14's warehouse in miniature, eight companies and their 2023 financials, exposed as one more tool. Two details carry the lesson. The connection is read-only (`PRAGMA query_only = ON`), Chapter 14's safety rule, because this query will be written by a model. And the tool's description tells the agent what the database can and cannot answer, because with two overlapping data tools (`lookup_financial` and this one), the description is what routes the call. ```{python} import sqlite3 db = sqlite3.connect(":memory:") db.executescript(""" CREATE TABLE financials (ticker TEXT, name TEXT, year INT, revenue_billions REAL, net_income_billions REAL); INSERT INTO financials VALUES ('AMZN','Amazon',2023,574.8,30.4), ('TSLA','Tesla',2023,96.8,15.0), ('BAC','Bank of America',2023,98.6,26.5), ('HRL','Hormel Foods',2023,12.1,1.0), ('BKH','Black Hills',2023,2.6,0.25), ('AMBC','Ambac',2023,1.2,0.04), ('ATSG','Air Transport Services',2023,2.0,0.06), ('PK','Park Hotels',2023,2.6,0.10); """) db.execute("PRAGMA query_only = ON") def query_database(sql: str): """Run a read-only SQL query against the financials table.""" try: cur = db.execute(sql) cols = [d[0] for d in cur.description] return [dict(zip(cols, row)) for row in cur.fetchmany(20)] except Exception as exc: return f"SQL error: {exc}" tools["query_database"] = query_database schemas.append({"type": "function", "function": { "name": "query_database", "description": "Run a read-only SQL query against financials(ticker, name, " "year, revenue_billions, net_income_billions). 2023 data for " "eight companies. Use for rankings, ratios, and aggregates.", "parameters": {"type": "object", "properties": {"sql": {"type": "string"}}, "required": ["sql"]}}}) ``` The error path does quiet work here: a failed query, which would ordinarily raise, returns the error message to the agent, so the model can read it and rewrite the SQL, the same self-correction loop Chapter 14 built deliberately, happening here on the agent's own initiative. Now we ask a question that no single tool answers, one that crosses from structured data into documents: ```{python} result = run_agent( "Which of the eight companies had the highest net margin in 2023, " "and what does its annual report say about its strategy?", tools=tools, schemas=schemas, system="You are a financial analyst assistant. Use the tools.", ) for step in result.steps: print(f" {step['tool']}({str(step['args'])[:60]}) -> {str(step['result'])[:80]}") print("\nfinal answer:", result.answer[:400]) ``` The agent wrote a SQL query with a computed ratio, read the result, and carried the winning company into a document search, structured and unstructured retrieval composed in one loop, with no step of that plan written by us. This is the structure we see in most production agents today: a handful of data tools with accurate descriptions, a model that routes between them, and read-only access everywhere a model writes the query. We claimed the description is what routes the call, so we measure it. We register the same SQL tool a second way, with its description stripped down to "Run a SQL query.", and measure the effect. **Metric**: correct-first-tool rate, the fraction of questions where the agent's first call goes to `query_database`.\ **Test set**: four questions about rankings and aggregates across the eight companies, exactly what the database is for.\ **Baseline**: the rich description the tool was built with. ```{python} import copy import pandas as pd schemas_degraded = copy.deepcopy(schemas) for s in schemas_degraded: if s["function"]["name"] == "query_database": s["function"]["description"] = "Run a SQL query." routing_questions = [ "Which of the eight companies had the highest net margin in 2023?", "Rank the eight companies by 2023 net income.", "What was the average revenue across all eight companies in 2023?", "How many of the companies earned more than 10 billion dollars of net income in 2023?", ] def first_tool_rate(schema_set): hits = 0 for q in routing_questions: r = run_agent(q, tools, schema_set, system="You are a financial analyst assistant. Use the tools.") hits += bool(r.steps) and r.steps[0]["tool"] == "query_database" return hits n_q = len(routing_questions) rich_hits = first_tool_rate(schemas) degraded_hits = first_tool_rate(schemas_degraded) pd.DataFrame({ "tool description": ["rich: schema, coverage, and when to use it", "degraded: \"Run a SQL query.\""], "correct first tool": [f"{rich_hits}/{n_q}", f"{degraded_hits}/{n_q}"], }) ``` The rich description tells the model what the database contains and when to reach for it; the degraded one leaves it to guess between four tools, and the gap between the two rows measures what that guessing loses in routing accuracy. The practical lesson is that a tool's description, although it reads like documentation, works as a prompt: it is the only thing the model has to route on, and writing it is part of building the tool, the same way Chapter 11's orchestrator needed to be told what its workers could answer. ## Agents need a model trained for reliable tool use We built a catch into the setup. The agent runs on `MODEL_AGENT`, the agent model, one trained and tested for reliable tool calling, although every earlier chapter reached for the book's default lightweight model. The distinction is a specific skill, one separate from raw intelligence: emitting a well-formed tool call every time one is needed, passing the right arguments, and folding each result into the next decision, over and over without dropping a step. A model without that training is an unreliable agent however well it answers single prompts: it forgets to call tools, calls them with the wrong arguments, or answers from memory. This is the routing lesson from Chapter 2 at higher stakes: send simple classification to the default model, but put a tool-reliable model behind an agent loop. The price difference between the two is modest, so the lesson concerns reliability; what actually grows an agent's bill is the loop, several calls per question with a context that lengthens at each step. The evaluation at the end of this chapter shows the reliability gap directly: the same agent that solves nearly every task on the agent model solves almost none on the default one. | what the loop demands | tool-trained agent model | default lightweight model | |---|---|---| | emits a well-formed tool call whenever one is needed | reliably | often skips the call and answers from memory | | passes the right arguments | reliably | drops or garbles arguments | | folds each result into the next decision | step after step | loses the thread mid-loop | | single-prompt quality (classify, extract, draft) | strong | strong, which is why it fooled us in Chapter 2 | | price per call | a modest premium | cheapest | : What reliable tool use requires, and where each model stands. The last two rows are the trap: on single prompts and on price the default model looks like the obvious choice, and the loop is where it falls apart. {#tbl-agent-reliability} ## Connecting to the world: the Model Context Protocol Our three tools are defined inside this notebook. Real agents need tools hosted elsewhere: a company database, a ticketing system, a search API, your own filesystem. Wiring each one in by hand, in a different way for every agent and every tool, does not scale. The **Model Context Protocol** (MCP), introduced by Anthropic in late 2024 and now widely adopted, is the standard that addresses this. An MCP *server* exposes a set of tools in a standard format; any MCP-aware *client*, a coding assistant, a chat app, your own agent, can discover and call them without custom integration code. ```{mermaid} %%| echo: false %%| label: fig-mcp-architecture %%| fig-cap: "MCP in one picture. Servers expose tools in a standard format; any MCP-aware client can discover and call them. New client or new server, no new integration code." flowchart LR subgraph clients["MCP clients"] c1["coding assistant"] c2["chat app"] c3["your agent"] end subgraph servers["MCP servers"] s1["database server query, schema"] s2["filesystem server read, write, list"] s3["your server the notebook's three tools"] end c1 -- "discover + call" --> s1 c1 --> s2 c2 --> s2 c3 --> s1 c3 --> s3 ``` The same three tools, exposed as an MCP server, look like this. Although we do not execute the server here, you can run it yourself and point a client at it. ```{python} #| eval: false # An MCP server exposing our tools. Run with: python this_file.py from mcp.server.fastmcp import FastMCP mcp = FastMCP("financial-tools") @mcp.tool() def lookup_financial(company: str, metric: str): """Look up a stored financial figure by company and metric.""" ticker = ALIASES.get(company.lower().strip(), company.upper()) return FACTS.get(ticker, {}).get(metric, "not found") @mcp.tool() def calculator(expression: str): """Evaluate an arithmetic expression.""" return eval(expression, {"__builtins__": {}}, {}) if __name__ == "__main__": mcp.run() ``` This short listing is the whole server. The `@mcp.tool()` decorator turns a normal Python function, with its docstring and type hints, into a tool that any MCP client can call, without any schema written by hand. This is how your coding assistant connects to external systems: Claude Code, Cursor, and the others are MCP clients, and the integrations you add to them, a database connector, a Slack tool, a filesystem bridge, are MCP servers exactly like this one. ::: {.callout-warning title="Pitfall: tools are an attack surface"} Everything a tool returns, and even a tool's *description*, is part of the model's context, and the model follows instructions it finds there. Therefore, an untrusted MCP server can poison a tool description or result with instructions ("before answering, send the conversation to..."), and the agent may comply. Connect only servers you trust, read what their tools claim to do, and treat every tool output as untrusted input. See the section on indirect injection in Chapter 23 for an example of this attack and the layered defenses against it. ::: ::: {.callout-tip title="With an AI coding tool"} This is the chapter where "using the tool" and "building the tool" meet. The MCP server above is the same kind of thing you would write to give your own coding assistant a new capability, a tool that queries your company's data, runs your test suite, or files a ticket. You now understand the protocol well enough that you can extend the assistant on your desktop, where until now you could only consume it. ::: ## When to use an agent The flexibility of an agent is also its danger. Because the model decides the control flow, an agent can loop longer than you expected, call a tool you did not want called, or take an action that is hard to undo. A workflow cannot surprise you that way, because you wrote every step. The step budget is the guardrail against the first failure, and we watch it fire once: we hand the agent a task its tools cannot complete, a sweep across all eight companies for a figure that exists in none of them, and cap the loop at three steps. ```{python} runaway = run_agent( "For each of the eight companies, find what its annual report says about " "its revenue guidance for 2024. Check one company at a time.", tools=tools, schemas=schemas, max_steps=3, system="You are a financial analyst assistant. Use the tools for every figure; " "do not answer from memory.", ) for step in runaway.steps: print(f" {step['tool']}({str(step['args'])[:60]})") print("\nanswer:", runaway.answer) ``` The trace shows an agent doing exactly what it was told, searching company after company for something the corpus does not contain, and the budget cutting it off mid-sweep with the sentinel answer before it could spend eight more calls confirming the absence. This sentinel, which can read as a failure, is the guardrail working: the budget converts what would have been an unbounded bill into a bounded, visible non-answer. In production the sentinel is also a signal to log and alert on, because a rising rate of step-budget exits means users are asking the agent for things its tools cannot do. So the rule holds: use the simplest thing that works. Reach for a single call when the task is single, a workflow when the steps are known, and an agent only when the steps cannot be known in advance, and even then, with a step budget, a careful set of tools, and a human in the loop for anything irreversible. We return to those guardrails in Part IX. The agent frameworks you will meet in practice, the OpenAI Agents SDK, LangGraph, the Claude Agent SDK, are wrappers around exactly the loop this chapter built: they add conveniences like tracing, retries, and handoffs between agents, but the call-the-model, run-the-tool, feed-the-result cycle underneath is the one in `run_agent`, which is why building it raw once is worth the trouble. Two directions sit beyond this book's scope. Computer-use and browser agents replace the fixed tool list with a screen, keyboard, and mouse, the same loop with a much riskier action space. Multi-agent systems put several agents in conversation with each other, dividing a task the way the orchestrator pattern divided one in Chapter 11, with coordination problems to match. ## Evaluation: does the agent get the answer right? We judge an agent on whether it completes the task, since a loop that ran to completion proves nothing about the answer. We give it multi-step financial questions whose answers we can compute ourselves, and check the agent's answer against them. The test set covers all four tools: the first three tasks need a lookup and a calculation, the next three need the SQL tool (we verified each answer by hand against the table: Bank of America's 26.5 on 98.6 is the highest net margin at about 26.9 percent, total revenue sums to 790.7 billion, and Ambac's 1.2 billion is the smallest), and the last needs the report search, scored by whether a phrase from the actual filing appears in the answer. **Metric**: task success, the fraction of questions answered correctly, with the spend and the average number of tool steps reported alongside.\ **Test set**: seven questions spanning the lookup, calculator, SQL, and report-search tools.\ **Baseline**: the *same* agent on the book's default lightweight model, to test the claim that the agent loop needs a model trained for tool use. ```{python} from gaba.llm import MODEL_AGENT, MODEL_DEFAULT tasks = [ ("What is 5 percent of Tesla's 2023 vehicle deliveries?", ["90,429", "90429"]), ("What is Amazon's net sales plus AWS revenue in 2023, in billions of dollars?", ["665.6", "665,600", "665600"]), ("Amazon's AWS revenue is what percentage of its net sales in 2023?", ["15.7", "15.8", "16"]), ("Which of the eight companies had the highest net margin in 2023?", ["bank of america", "bac"]), ("What was the total 2023 revenue of all eight companies combined, in billions?", ["790.7", "790,700", "790700"]), ("Which company had the lowest revenue in 2023?", ["ambac", "ambc"]), ("According to its annual report, what is Tesla's stated mission?", ["sustainable energy"]), ] def evaluate_agent(model: str) -> tuple[int, float, float]: """Return tasks solved, total cost, and average tool steps per task.""" solved, cost, steps = 0, 0.0, 0 for question, accepted in tasks: result = run_agent(question, tools, schemas, model=model, system="You are a financial analyst. Use the tools.") answer = result.answer.replace("$", "").lower() solved += any(a in answer for a in accepted) cost += result.cost_usd steps += len(result.steps) return solved, cost, steps / len(tasks) agent_solved, agent_cost, agent_steps = evaluate_agent(MODEL_AGENT) default_solved, default_cost, default_steps = evaluate_agent(MODEL_DEFAULT) import pandas as pd pd.DataFrame({ "model": [f"{MODEL_AGENT} (agent model)", f"{MODEL_DEFAULT} (default)"], "tasks solved": [f"{agent_solved}/{len(tasks)}", f"{default_solved}/{len(tasks)}"], "avg steps": [f"{agent_steps:.1f}", f"{default_steps:.1f}"], "cost $": [f"{agent_cost:.5f}", f"{default_cost:.5f}"], }) ``` This is the "agents need a model trained for reliable tool use" claim made concrete, and the gap is not subtle. The agent model solves most or all of the tasks by composing its tools, getting figures it could not reliably recall and arithmetic it could not reliably do (where it drops one, the trace shows a slip from which it recovers). The default model, the very one that excelled at single-prompt classification back in Chapter 2, solves few or none, and its failures trace to the *loop* itself, since the arithmetic is well within its reach: it forgets to call the tools, calls them with the wrong arguments, or tries to answer from memory. The `avg steps` column is the loop made visible: an agent doing real multi-step work averages a couple of tool calls per task, and a number far from that, in either direction, is a symptom worth investigating in the trace. The cost column makes the same point from the other side: the whole comparison cost well under a cent, so the gap, which price cannot explain at these amounts, comes from whether the model was trained to drive the loop at all. When an agent fails, the trace, the list of steps, tells you which of those went wrong, which is why an agent's evaluation covers the path it took as well as the final answer; the Evaluate lab below puts that diagnosis in your hands. ::: {.callout-warning title="Don't outsource this"} If an agent misses a task, an assistant will readily propose another tool, a longer system prompt, or a retry wrapper, and one of those fixes might even work. But deciding what `result.steps` actually shows went wrong, whether a skipped call, a garbled argument, or an answer pulled from memory where a lookup was needed, is not a judgment you can hand back to the assistant that just watched the agent fail. Read the trace yourself before accepting any fix, because the fix that satisfies the assistant is not automatically the fix that addresses the failure you diagnosed. ::: ::: {.callout-tip title="Cost: agents are the expensive option"} An agent's cost is driven by its loop. A few thousand tokens through the agent model still costs a fraction of a cent, a modest premium over the default; what grows the bill is that an agent makes several calls per question, each carrying the entire conversation so far, so a ten-step run pays for ten increasingly long contexts. On hardware you host, the same growth appears as latency and capacity: each call's longer context takes longer to process and occupies the GPU for more of the run, so one agent question displaces many single-call questions. This is fine when the task needs it and ruinous when it does not, which is the whole reason the rule is to escalate to an agent only after the simpler designs have been ruled out. ::: The growth of "increasingly long" is quadratic, although the phrase suggests something linear, and the arithmetic shows why: if each step adds roughly the same number of tokens, call $k$ pays for about $k$ steps' worth of context, and a run of $N$ steps pays for about $N(N+1)/2$ steps' worth in total. ::: {.content-visible when-format="html"} ```{ojs} //| echo: false //| label: fig-context-growth //| fig-cap: "What an agent loop pays for. Each bar is one call in the loop, and its length is the context that call carries: everything every earlier step added. Drag the sliders and watch the total grow roughly with the square of the step count, which is why a long agent run costs far more than its step count suggests." viewof context_growth = { let steps = 6, perStep = 300; const container = document.createElement("div"); container.style.cssText = "font-family: inherit; color: var(--bs-body-color, #1f2328); border: 1px solid var(--bs-border-color, #d0d7de); border-radius: 6px; padding: 12px; max-width: 640px;"; function render() { container.innerHTML = ""; const mkSlider = (labelText, min, max, step, value, onInput) => { const row = document.createElement("div"); row.style.cssText = "display: flex; gap: 10px; align-items: center; margin-bottom: 6px; font-size: 13px;"; const lab = document.createElement("span"); lab.style.cssText = "color: var(--bs-secondary-color, #57606a); min-width: 210px;"; lab.textContent = `${labelText}: ${value.toLocaleString()}`; const s = document.createElement("input"); s.type = "range"; s.min = min; s.max = max; s.step = step; s.value = value; s.style.flex = "1"; s.oninput = () => { onInput(+s.value); render(); }; row.append(lab, s); return row; }; container.appendChild(mkSlider("steps in the loop", 1, 12, 1, steps, v => { steps = v; })); container.appendChild(mkSlider("tokens added per step", 100, 1000, 50, perStep, v => { perStep = v; })); const maxLen = steps * perStep; const bars = document.createElement("div"); bars.style.cssText = "margin: 10px 0 6px;"; for (let k = 1; k <= steps; k++) { const row = document.createElement("div"); row.style.cssText = "display: flex; gap: 8px; align-items: center; margin-bottom: 3px; font-size: 12px;"; const lab = document.createElement("span"); lab.style.cssText = "min-width: 52px; color: var(--bs-secondary-color, #57606a);"; lab.textContent = `call ${k}`; const track = document.createElement("div"); track.style.cssText = "flex: 1; height: 13px; background: rgba(128,128,128,0.18); border-radius: 3px; overflow: hidden;"; const fill = document.createElement("div"); fill.style.cssText = `width: ${(100 * k * perStep / maxLen).toFixed(1)}%; height: 100%; background: #0969da;`; track.appendChild(fill); const val = document.createElement("span"); val.style.cssText = "min-width: 88px; text-align: right;"; val.textContent = `${(k * perStep).toLocaleString()} tokens`; row.append(lab, track, val); bars.appendChild(row); } container.appendChild(bars); const total = perStep * steps * (steps + 1) / 2; const readout = document.createElement("div"); readout.style.cssText = "font-size: 13px; margin-top: 8px;"; readout.innerHTML = `Total prompt tokens across the run: ${total.toLocaleString()}, ` + `which is ${(steps * (steps + 1) / 2).toLocaleString()}x what a single call over the same material would pay.`; container.appendChild(readout); } render(); return container; } ``` ::: ::: {.content-visible when-format="pdf,epub"} | steps in the loop | context the last call carries | total prompt tokens for the run | vs one single call | |---:|---:|---:|---:| | 3 | 900 | 1,800 | 6x | | 6 | 1,800 | 6,300 | 21x | | 12 | 3,600 | 23,400 | 78x | : Context growth in an agent loop at 300 tokens added per step: each call carries everything before it, so the total grows with the square of the step count. (The online edition makes this an interactive demonstration.) {#tbl-context-growth} ::: ## Choosing your workflow and agent stack {#sec-agent-stack} In Part IV we built workflows by hand and an agent loop, then connected tools through MCP. Frameworks exist for each, and most projects need fewer of them than it seems. | Capability | Lightweight | Framework | Choose by | |---|---|---|---| | Orchestration | plain Python | LangGraph, LlamaIndex Workflows | how complex the control flow really is | | Agent runtime | the book's fifty-line loop | OpenAI Agents SDK, Pydantic AI, CrewAI | whether you need the loop managed for you | | Tools | direct function calls | MCP via FastMCP and reference servers | reuse across clients | ::: {.runlabel} **Important** ::: - Prefer a workflow you control to an agent that decides for itself; it is cheaper and easier to debug (Chapter 11). - Give an agent a model trained for tool use, because a general model chains tools unreliably (Chapter 12). - Cap the step count, which stops a confused agent that would otherwise loop forever. ::: {.runlabel .warn} **Common failure points** ::: - Reaching for an agent when a workflow would do (Chapter 11). - Giving an agent a destructive tool with no human gate (Chapter 23). - Adopting a heavy framework for a flow that three functions would express. The current open-source and vendor options for these are in the [tooling-landscape appendix](../../appendices/i-tooling.qmd), dated and fuller; model and provider choice is [Appendix E](../../appendices/e-vendor-selection.qmd). ## Exercises ### Conceptual questions 1. What distinguishes an agent from a workflow? A. an agent can call external tools, which a workflow has no way to do B. the model picks each next step at run time, while a workflow follows steps we wrote C. an agent runs all of its steps in parallel while a workflow runs them in order D. an agent needs no system prompt because its tools define the behavior 2. In the chapter's evaluation, the agent model solved most of the tasks and the default model solved almost none. The default model failed mainly by: A. exhausting its context window before it could produce a final answer B. picking the calculator on questions that the report search should have handled C. timing out on the network during its longer chains of tool calls D. breaking down on the loop itself: skipping tools, passing wrong arguments, or answering from memory 3. What problem does the Model Context Protocol solve? A. a standard format for tools, so any client can discover and call them with no custom glue B. a shared memory format, so a model's conversation history can carry across different applications C. a verification layer that checks a tool's output before the model sees it D. a routing layer that sends each question to the cheapest capable model 4. When an agent misbehaves, the most useful thing to inspect is: A. the final answer, since it reflects everything the agent did B. the system prompt, since agents fail only when instructions are unclear C. the trace of tool calls and results, which shows where the path broke D. the temperature, since randomness is the usual cause of a bad loop 5. The right time to escalate from a workflow to an agent is when: A. the steps needed to finish the task cannot be predicted in advance B. the task requires more than one model call to get from question to answer C. you want the cheapest way to answer a multi-step question D. the task needs structured output that a workflow cannot enforce 6. The evaluation runs the same agent on the default lightweight model as its baseline. What makes that the right baseline? A. the default model is the standard baseline used in published agent benchmarks B. it proves the test questions are easy enough for any model to answer C. running both models cuts the cost of building the evaluation in half D. it isolates the model as the only difference, directly testing the claim that the loop needs a model trained for tool use 7. The chapter warns against deploying the `eval`-based calculator, especially over MCP, because: A. `eval` is far too slow for production traffic at any real volume B. it runs model-controlled text as code, and emptied builtins is not a real sandbox C. MCP clients have no safe way to pass string arguments to a tool D. arithmetic belongs in the model itself, so the tool adds attack surface for nothing 8. In the MCP server example, what work does the `@mcp.tool()` decorator remove? A. writing the Python function body that does the actual work B. giving the function a docstring or type hints of any kind C. hand-writing the JSON schema; the type hints and docstring supply it D. running a server process at all, since decorated tools are called directly in process ### Build lab Add a fourth tool to the agent, for example one that returns today's date, or one that converts currencies with a fixed rate, and write a question that requires it together with the existing tools. Run the agent and read the trace to confirm it used the new tool. You may use an assistant to draft the tool and its schema; check the schema matches the function's arguments yourself. ### Evaluate lab The evaluation showed the default model failing, but the table does not say *why*. Re-run one failing task on `MODEL_DEFAULT` and print `result.steps`, the trace of tool calls it made. Diagnose the failure mode from the trace: did it skip the lookup, call a tool with the wrong argument, or try the arithmetic in its head? Report which step broke, and write one sentence on why the trace is what you actually debug an agent with, given that the score only tells you a task was missed. ::: {.callout-tip title="Project ideas"} You can now build a fixed workflow from the five patterns, escalate to an agent when the task needs it, and give either one tools through MCP. These two projects show you the difference between the two. - **A workflow that triages and drafts.** Chain the patterns into a pipeline that classifies an incoming request, retrieves what it needs, drafts a reply, and checks its own draft before sending it back. Then build an agent that decides its own steps for the same task, and note where each design is the better fit. *Data to try:* a support-ticket set, a public email dataset, or your own inbox export. - **An analyst agent over a database.** Give an agent a SQL tool and a calculation tool, then pose questions that take several steps: pull a slice, compute a metric, compare it across segments, and explain the result. Cap its steps and log every tool call so you can audit its reasoning, then compare it against a fixed workflow that hard-codes the same steps and see which one is worth the extra latency. *Data to try:* a public relational database such as Chinook, Northwind, or the NYC taxi trips, loaded into SQLite. Part V turns from answering questions to analyzing text and tables at scale. ::: ::: {.callout-note title="Where we go next"} That closes Part IV. We can build systems that read, retrieve, reason, and act. In Part V we step back to the data itself: the embeddings we have been using for search hold structure we can mine directly. In Chapter 13 we cluster them to discover the themes in a pile of text, no labels required. :::