3 Structured outputs

From prose to JSON to pandas

Our triage system from Chapter 2 has a hidden flaw: it asks the model for a category, and if it does not get a single clean word, it falls back to other, so errors that should be reported are silently recorded as valid data. In this chapter, we fix this by having the model return data that conforms to a Pydantic model that describes the exact structure of the output we want. Once the output is typed, we can load it directly into a dataframe and finally score the system against the labels for the first accuracy measurement of our triage.

Setup for this chapter

Run in the gaba-core environment. This chapter introduces gaba.llm.call_structured, which uses the Instructor library to turn a model’s reply into a validated Pydantic object. It is already installed in gaba-core.

from dotenv import load_dotenv
load_dotenv()

import pandas as pd

3.1 Tools in this chapter

Tool	Why we use it here	Alternatives	Trade-off
Pydantic	describes the output structure as a typed class and validates data against it	Python dataclasses, jsonschema, hand-written checks	enforced types against a little boilerplate
Instructor	wraps the model call to return a validated Pydantic object, retrying when validation fails	the providers’ native structured-output modes	one behavior across every provider against an occasional retry cost

The trade-off against native structured outputs is weighed later in this chapter, and the tooling-landscape appendix lists the options.

3.2 Where loose parsing breaks

Recall the parser from Chapter 2: take the model’s text, strip it, lowercase it, and if the result is not in our category list, call it other. That works only as long as the model returns exactly one bare category and nothing else. The moment we ask for anything more, it breaks down.

Here is the result of asking for a category and a short reason in one free-text call, which is a completely reasonable thing to want:

from gaba.llm import call_llm
from gaba.data import TICKET_CATEGORIES

ticket = "The blender arrived shattered, glass everywhere. I want my money back."

# Ask for a category and a reason together, as free text.
loose = call_llm(
    prompt=ticket,
    system=(
        "Classify this support ticket. Reply in the form: category - reason. "
        "Choose the category from: " + ", ".join(TICKET_CATEGORIES)
    ),
)
print("model said:", loose.text)

# Now apply Chapter 2's parser to pull out the category.
parsed = loose.text.strip().lower()
category = parsed if parsed in TICKET_CATEGORIES else "other"
print("parsed category:", category)

model said: damaged_item - The item arrived broken and the customer is requesting a refund.
parsed category: other

The model answered sensibly, yet the parser failed: the reply was a whole sentence where the parser expected a bare category, so it did not match the list and silently became other. A real category was lost without any error being raised. Run this across ten thousand tickets and you would have a column full of other that resembles data but is in fact corruption.

The fragility extends beyond this naive parser to the format of the answer itself. Ask for several fields at once and the model obliges, but in whatever format it prefers.

from gaba.data import load_tickets

ask = ("Give the category (one of: " + ", ".join(TICKET_CATEGORIES) + "), "
       "the priority (low, medium, or high), and a one-line reason.")
for text in load_tickets()["text"].head(3):
    print(call_llm(prompt=text, system=ask).text.strip())
    print("-" * 8)

billing_problem, high, Customer was charged twice for a single order.
--------
shipping_delay, high, The customer is frustrated with repeated delivery delays and wants to cancel.
--------
damaged_item, high, The blender jar arrived shattered.
--------

Each answer is sensible, but the model chooses the format, and rarely the same format twice: a comma-separated line here, the dash-separated one above. To turn either into a dataframe you split on whichever separator the model happened to use, then hope the reason never contains one, and that the next model or the next prompt tweak does not switch the format again. The fields are present, but extracting them reliably, in the same way on every reply, is the fragile step.

The deeper problem is that the contract between us and the model existed only as an English sentence in the prompt (“reply in the form…”), and English instructions are not enforced. Because an assumed structure carries no enforcement, the structure of the answer must be something the code checks.

3.3 Describing the structure we want: Pydantic

Pydantic lets us write that structure down as a class. Each field has a type, and Pydantic validates any data we put into it against those types. If the data does not fit, it raises an error, which keeps bad values from passing through.

Here is the structure we actually want from a ticket: a category drawn from our fixed list, a priority that is one of three levels, and a one-sentence reason.

from enum import Enum
from typing import Literal
from pydantic import BaseModel, Field

# Build an Enum from our category list so the schema and the data agree on
# exactly which categories exist. The model will be allowed to choose only
# these values, nothing else.
TicketCategory = Enum("TicketCategory", {c: c for c in TICKET_CATEGORIES}, type=str)


class TicketAnalysis(BaseModel):
    """The structured result we want for every ticket."""

    category: TicketCategory
    priority: Literal["low", "medium", "high"]
    reason: str = Field(description="one short sentence explaining the choice")

The benefit is that invalid data cannot pass as valid. If something tries to set a category that is not on the list, Pydantic refuses it.

from pydantic import ValidationError

# A good value validates fine.
ok = TicketAnalysis(category="damaged_item", priority="high", reason="Glass arrived broken.")
print("valid:", ok.category.value, ok.priority)

# A bad category is rejected, loudly, instead of slipping through as "other".
try:
    TicketAnalysis(category="please_help_me", priority="high", reason="x")
except ValidationError as e:
    print("\nrejected, as it should be:")
    print(e.errors()[0]["msg"])

valid: damaged_item high

rejected, as it should be:
Input should be 'billing_problem', 'refund_request', 'shipping_delay', 'damaged_item', 'wrong_item', 'account_access', 'technical_issue', 'product_question', 'cancellation', 'subscription_change', 'feature_request', 'complaint', 'praise', 'fraud_report' or 'other'

The loud rejection is the point: where the loose parser hid the mistake, the validator reports it. The remaining step is getting the model to produce data in this structure in the first place.

3.4 Typed output from a model

The model’s output has to leave the model and enter our code, and code needs structure it can rely on. JSON is the common format for that, a text format of keys and values that every programming language parses without extra work. The alternative is to let the model reply in sentences and parse them ourselves with regular expressions or string-splitting, which works until the phrasing shifts and then fails quietly, the flaw the previous section showed. JSON fixes the format of the output but leaves its meaning unchecked: it ensures the fields are present and typed, which is why we still validate the values. Other structured formats exist, such as XML (Extensible Markup Language) and YAML (a human-readable data format), but JSON is what the model providers support directly, so it is what we use.

Modern models can return JSON on request, and several libraries make them return JSON that matches a Pydantic schema specifically. We use Instructor. It sends the model our schema, asks for a JSON object that fits, and validates the reply against the Pydantic model. If validation fails, it sends the error back to the model and asks again. What we get back is a validated TicketAnalysis with no string left for us to parse.

We wrap this in gaba.llm.call_structured, which is like call_llm but takes a schema and returns the validated object as well as the usual cost bookkeeping.

With an AI coding tool

Drafting a Pydantic schema from an example of the JSON you want is a good use of an AI coding tool. Paste a sample object, ask for the model, then read every field: is the type right, is it required or optional, is the set of allowed values complete? The reading is the part that matters.

from gaba.llm import call_structured

result = call_structured(ticket, TicketAnalysis)

analysis = result.data  # a validated TicketAnalysis instance
print("category:", analysis.category.value)
print("priority:", analysis.priority)
print("reason:  ", analysis.reason)
print(f"\ncost: ${result.cost_usd:.6f}")

category: damaged_item
priority: high
reason:   The item arrived broken and the customer is requesting a refund.

cost: $0.000052

The same ticket that broke the loose parser now returns as a typed object with category guaranteed to be one of our fifteen, priority guaranteed to be one of three, and a reason that we can read or ignore. There is no parsing step left to fail.

Don’t outsource this

A tool can generate the schema, but it cannot decide for your business which fields are required and which are optional, or what should happen if a real document does not have a field you specified. Forcing the model to fill a field that is not in the source is how you manufacture hallucinations. These are your decisions, and we come back to them when extracting from real filings in Chapter 5.

3.4.1 Native structured outputs

Instructor’s validate-and-retry loop is not the only way to get schema-conforming output. The major providers have all offered native structured outputs since 2024-25: you pass a JSON schema in the request (response_format with a json_schema type in strict mode in the OpenAI protocol, or responseSchema in Google’s Gemini), and the provider constrains the model’s decoding with a grammar built from the schema. The model cannot emit a token that would break the JSON structure, so the output is guaranteed-valid JSON on the first try, with no retry loop and no reask cost.

This book uses Instructor for two reasons. First, for consistency: we route every call through OpenRouter to reach many different models, and native strict mode is implemented unevenly across these models, while Instructor’s reask loop works the same way everywhere. Second, for semantics: constrained decoding guarantees format while leaving meaning unchecked. A grammar can force amount to be a number, although it cannot check that the amount is positive, that a date is not in the future, or any other business rule. Pydantic validators can do that, and Instructor feeds their failures back to the model. The two approaches compose: format from the grammar, meaning from the validators.

If you are committed to a single provider, use its native structured outputs as the base layer. It is faster, cheaper, and strictly more reliable at producing parseable JSON. Add semantic validation on top only where your fields have rules that a grammar cannot express.

3.4.2 Validators carry the business rules

In a refund pipeline, the simplest rule a grammar cannot express is that a refund must be a positive dollar amount. A schema can force refund_usd to be a number, but only a validator can insist on its sign. In Pydantic, that rule is a field_validator that rejects bad data no matter where it came from.

from pydantic import field_validator


class RefundDecision(BaseModel):
    """A refund extracted from a ticket, with one business rule attached."""

    refund_usd: float = Field(description="the refund amount in USD")

    @field_validator("refund_usd")
    @classmethod
    def must_be_positive(cls, v: float) -> float:
        if v <= 0:
            raise ValueError("refund_usd must be a positive dollar amount")
        return v


# The rule fires on any invalid payload, model-made or hand-made.
try:
    RefundDecision(refund_usd=-20.0)
except ValidationError as e:
    print("rejected:", e.errors()[0]["msg"])

rejected: Value error, refund_usd must be a positive dollar amount

Because Instructor feeds validation errors back to the model, this rule does more than reject: it triggers a reask, at our expense. Observing this requires a ticket designed to provoke the failure, one whose literal text contains a negative amount, and a way to count attempts. Instructor exposes hooks that fire on every raw completion, so counting completions counts attempts.

from gaba.llm import get_structured_client, cost_estimate, MODEL_DEFAULT

# Each completion fires the hook once, so the list length is the attempt count
# and each entry is what that attempt cost.
attempt_costs = []
def record_attempt(response) -> None:
    attempt_costs.append(cost_estimate(
        MODEL_DEFAULT, response.usage.prompt_tokens, response.usage.completion_tokens))

client = get_structured_client()
client.on("completion:response", record_attempt)

# The trap: a faithful extraction of this ticket is a negative number,
# which the validator will refuse.
trap_ticket = ("Your agent promised me a refund of -$20 on yesterday's call. "
               "Please process it.")
res = call_structured(
    prompt=trap_ticket,
    schema=RefundDecision,
    system="Extract the refund amount the customer says they were promised.",
    max_retries=3,
)
client.off("completion:response", record_attempt)

print(f"validated refund_usd: {res.data.refund_usd}")
print(f"attempts: {len(attempt_costs)}")
print(f"first attempt cost: ${attempt_costs[0]:.6f}; "
      f"reask cost: ${sum(attempt_costs[1:]):.6f}")

validated refund_usd: 20.0
attempts: 2
first attempt cost: $0.000022; reask cost: $0.000078

The object that comes back satisfies the rule either way; what varies is the path. When the model’s first reply faithfully copies the negative number, the validator rejects it, Instructor sends the error back, and the attempt count above reads two or more, with the reask cost as the price of the correction. When the model silently fixes the sign on its own, the count reads one and the reask cost is zero. In our experience either outcome occurs depending on the model and the phrasing, which is exactly what the validator is for: it is the difference between hoping the model corrects bad data and guaranteeing that bad data cannot pass.

3.5 Triage, now typed

With call_structured and the schema, the triage loop becomes both simpler and safer. Every result is a typed object, so we read .category and .priority as attributes, with no text left to parse.

from gaba.data import load_tickets

tickets = load_tickets()

records = []
for _, row in tickets.iterrows():
    res = call_structured(
        prompt=row["text"],
        schema=TicketAnalysis,
        system="Classify the support ticket. Choose the single best category.",
    )
    records.append(
        {
            "ticket_id": row["ticket_id"],
            "category": res.data.category.value,
            "priority": res.data.priority,
            "cost_usd": res.cost_usd,
        }
    )

triage = pd.DataFrame(records)
triage.head(8)

	ticket_id	category	priority	cost_usd
0	T001	billing_problem	high	0.000056
1	T002	shipping_delay	high	0.000057
2	T003	damaged_item	high	0.000056
3	T004	wrong_item	high	0.000059
4	T005	account_access	high	0.000056
5	T006	product_question	low	0.000054
6	T007	cancellation	medium	0.000058
7	T008	subscription_change	medium	0.000055

Every row is clean: no fallbacks, no off-list values, nothing silently dropped. The category column is drawn from exactly the fifteen we defined, and we obtained a priority in the same call, with no second request.

3.6 When the schema demands what the source does not contain

The schema is a contract that binds both parties: mark a field required and the model must fill it on every ticket, including the tickets where the value simply is not there. The most common real extraction failure we know of is a schema that forces the model to answer a question the document never answers, a failure easy to misattribute to the model misreading the document. The “Don’t outsource this” warning above said the required-versus-optional choice is yours; here is what getting it wrong looks like.

Ten fictional tickets, of which exactly three mention an order ID. The gold list is written by hand: an ID where one exists, None where none does.

# Only the first three tickets contain an order ID.
id_tickets = [
    "Order ORD-58213 arrived with the wrong color headphones.",
    "Still waiting on ORD-91447, it has been two weeks now.",
    "I was double charged for order ORD-20786 and want one charge reversed.",
    "The app crashes every time I open the settings page.",
    "Your driver left the package in the rain and everything inside is soaked.",
    "How do I change the email address on my account?",
    "The subscription renewed even though I cancelled last month.",
    "The blender is much louder than the demo video suggested.",
    "I never received the discount code your newsletter promised.",
    "Two of the mugs in my delivery were chipped at the rim.",
]
gold_ids = ["ORD-58213", "ORD-91447", "ORD-20786",
            None, None, None, None, None, None, None]

Two schemas, identical except that the second allows order_id to be None.

class OrderRefRequired(BaseModel):
    order_id: str = Field(description="the order ID referenced in the ticket")


class OrderRefOptional(BaseModel):
    order_id: str | None = Field(
        None, description="the order ID referenced in the ticket, or null if the ticket gives none"
    )

Now both schemas extract from all ten tickets. Twenty calls, run in parallel.

from concurrent.futures import ThreadPoolExecutor


def extract_id(ticket: str, schema: type[BaseModel]):
    # A required field has a third possible outcome besides right and wrong:
    # if the model never satisfies the schema, Instructor gives up and raises.
    # We record that outcome instead of letting it stop the batch.
    try:
        res = call_structured(prompt=ticket, schema=schema,
                              system="Extract the order ID from this support ticket.")
        return res.data.order_id
    except Exception:
        return "<failed validation>"


with ThreadPoolExecutor(max_workers=8) as pool:
    required_ids = list(pool.map(lambda t: extract_id(t, OrderRefRequired), id_tickets))
    optional_ids = list(pool.map(lambda t: extract_id(t, OrderRefOptional), id_tickets))

comparison = pd.DataFrame({
    "ticket": [t[:44] + ("..." if len(t) > 44 else "") for t in id_tickets],
    "gold": gold_ids,
    "required_schema": required_ids,
    "optional_schema": optional_ids,
})
comparison

	ticket	gold	required_schema	optional_schema
0	Order ORD-58213 arrived with the wrong color...	ORD-58213	ORD-58213	ORD-58213
1	Still waiting on ORD-91447, it has been two ...	ORD-91447	ORD-91447	ORD-91447
2	I was double charged for order ORD-20786 and...	ORD-20786	ORD-20786	ORD-20786
3	The app crashes every time I open the settin...	NaN	<failed validation>	NaN
4	Your driver left the package in the rain and...	NaN	<failed validation>	NaN
5	How do I change the email address on my acco...	NaN		NaN
6	The subscription renewed even though I cance...	NaN	N/A	NaN
7	The blender is much louder than the demo vid...	NaN	<failed validation>	NaN
8	I never received the discount code your news...	NaN	<failed validation>	NaN
9	Two of the mugs in my delivery were chipped ...	NaN	12345	NaN

# Score only the seven tickets whose gold answer is "there is no ID".
no_id_rows = [i for i, g in enumerate(gold_ids) if g is None]
filled_anyway = sum(1 for i in no_id_rows
                    if required_ids[i] and required_ids[i] != "<failed validation>")
errored_out = sum(1 for i in no_id_rows if required_ids[i] == "<failed validation>")
correct_none = sum(1 for i in no_id_rows if optional_ids[i] is None)

print(f"tickets with no order ID: {len(no_id_rows)}")
print(f"required schema returned a value anyway: {filled_anyway}/{len(no_id_rows)}")
print(f"required schema failed outright:         {errored_out}/{len(no_id_rows)}")
print(f"optional schema correctly returned None: {correct_none}/{len(no_id_rows)}")

tickets with no order ID: 7
required schema returned a value anyway: 2/7
required schema failed outright:         4/7
optional schema correctly returned None: 7/7

Read the required_schema column on the no-ID rows. The model has to put something there, so it reaches for whatever satisfies the contract: a placeholder, an empty-looking token, or, worst of all, an invented ID with exactly the right format. An invented ORD- number is indistinguishable from a real one downstream, which is what makes this failure so expensive in practice; the occasional call that fails outright after its retries is the least damaging outcome, because that failure at least is visible. The optional schema gives the model a truthful way out, and it mostly takes it. The resulting rule: a field is required only when its absence from the source would itself be an error; everything else is str | None, and the Nones are data. We meet this rule again on real filings in Chapter 5.

3.7 Evaluation: accuracy against gold labels

The evaluation promised in Chapter 1 is now possible. In Chapter 2, we could report only cost, because we did not have any labels to check our answers against; we have those labels now.

Metric: classification accuracy, the fraction of tickets whose predicted category matches the gold label.
Test set: the thirty tickets, hand-labeled in tickets_labels.csv.
Baseline: the loose free-text parser from Chapter 2. The question is whether structuring the output improved the accuracy or only the format.

First, load the gold labels and score the structured triage.

from gaba.data import load_ticket_labels

labels = load_ticket_labels()
gold = dict(zip(labels["ticket_id"], labels["category"]))

triage["gold"] = triage["ticket_id"].map(gold)
structured_acc = (triage["category"] == triage["gold"]).mean()
print(f"structured accuracy: {structured_acc:.1%} "
      f"({(triage['category'] == triage['gold']).sum()}/{len(triage)})")

structured accuracy: 100.0% (30/30)

This score also revisits the comparison opened in Chapter 1, where the era-one keyword rules reached 73 percent on these same thirty tickets against these same labels, and their misses remain fixed until someone rewrites the rules by hand. This gap is the intent ladder of Chapter 1 made measurable, since the rules match exact words, while the generative system responds to what the customer meant, whatever words carried it, and it did so without knowing the queue’s vocabulary in advance.

For the baseline, we run the Chapter 2 loose parser over the same tickets and score it the same way, and we also count how many tickets it silently sent to “other”.

def loose_triage(text: str) -> str:
    """Chapter 2's parser: free-text category, fall back to 'other'."""
    r = call_llm(
        prompt=text,
        system=(
            "Reply with exactly one category, lowercase, nothing else: "
            + ", ".join(TICKET_CATEGORIES)
        ),
    )
    answer = r.text.strip().lower()
    return answer if answer in TICKET_CATEGORIES else "other"


loose_preds = [loose_triage(t) for t in tickets["text"]]
loose_acc = sum(p == g for p, g in zip(loose_preds, triage["gold"])) / len(loose_preds)
fell_to_other = sum(p == "other" for p in loose_preds)

print(f"loose-parser accuracy: {loose_acc:.1%}")
print(f"loose parser sent {fell_to_other} of {len(loose_preds)} tickets to 'other'")
print(f"structured accuracy:   {structured_acc:.1%}")

loose-parser accuracy: 100.0%
loose parser sent 0 of 30 tickets to 'other'
structured accuracy:   100.0%

On accuracy the two paths are nearly identical, and on these clean tickets the loose parser never even fell back to other. Although structuring did not make the model a better classifier, its effect appears in a property that accuracy cannot measure. Ask the model to name each ticket’s issue in its own words, with no fixed list, and the effect becomes visible.

from concurrent.futures import ThreadPoolExecutor


def freeform_label(text: str) -> str:
    # No list, no schema: the model phrases each ticket's issue however it likes.
    r = call_llm(prompt=text,
                 system="In two to four words, name the customer's main issue. "
                        "Reply with only the phrase, lowercase, no punctuation.")
    return r.text.strip().lower().rstrip(".")


with ThreadPoolExecutor(max_workers=8) as pool:
    freeform = list(pool.map(freeform_label, tickets["text"]))

print(f"free-form labels:  {len(set(freeform))} distinct for {len(freeform)} tickets")
print(f"schema categories: {triage['category'].nunique()} distinct for {len(triage)} tickets")

free-form labels:  30 distinct for 30 tickets
schema categories: 14 distinct for 30 tickets

import matplotlib.pyplot as plt

counts = [len(set(freeform)), triage["category"].nunique()]
fig, ax = plt.subplots(figsize=(5.2, 3.3))
bars = ax.bar(["free-form label", "schema category"], counts,
              color=["#9a9a9a", "#0969da"], width=0.55)
ax.bar_label(bars, padding=3)
ax.set_ylabel("distinct labels for 30 tickets")
ax.set_ylim(0, len(freeform) + 4)
ax.spines[["top", "right"]].set_visible(False)
plt.tight_layout()
plt.show()

Bar chart comparing distinct label counts for thirty tickets: a tall gray bar for free-form labels near thirty, and a much shorter blue bar for schema categories. — Figure 3.1: Distinct labels produced for the same thirty tickets. Asked for a free-text label, the model returns nearly one unique string per ticket, so nothing can be grouped or counted; the schema-constrained run collapses them to the bounded set of categories it is allowed to use.

The result is nearly one distinct string per ticket. The same shipping delay comes back as late delivery, then delayed package, then still waiting on my order, each a different string, and strings that cannot be grouped cannot be counted: ask how many shipping complaints arrived this week and the free-form column has no answer. The schema-constrained run is bounded by design, every value one of the categories we defined, so the thirty tickets collapse into a countable handful that totals, filters, and joins like any other column. Although structured output did not raise the accuracy here, it made the output usable as data. This usability is the primary argument for a schema, and it compounds with every field added: one free-text field fragments, three fields fragment three ways, and a parser written for the model’s current format breaks on its next.

flowchart TB
    subgraph loose["loose path: fails silently"]
        reply1["model reply<br/>(free text)"] --> parse["strip, lowercase,<br/>match against the list"]
        parse --> corrupt["off-list reply becomes 'other'<br/>and enters the dataframe"]
    end
    subgraph typed["structured path: fails loudly"]
        reply2["model reply<br/>(JSON)"] --> valid["Pydantic schema<br/>validation"]
        valid -- "fits" --> obj["typed TicketAnalysis"]
        valid -- "does not fit" --> err["loud ValidationError;<br/>Instructor reasks the model"]
    end
    loose ~~~ typed

Figure 3.2: The same model reply down two paths. The loose parser turns any unexpected reply into ‘other’ and lets it into the dataframe, which is silent corruption; schema validation returns either a typed object or a loud error that Instructor can retry, never a wrong answer that looks like a right one.

A high accuracy score also does not mean the task is unambiguous. Consider two tickets in our set. “Take me off my plan immediately and stop charging my card” sits between a cancellation and a subscription change. “I paid for express shipping and it arrived later… I want the shipping fee back” sits between a refund and a billing problem. Whether the model happened to match our label on these or not, the labels themselves are debatable, which puts a ceiling below 100 percent on accuracy against any single label set. We can list which tickets, if any, the model and our labels disagreed on:

misses = triage[triage["category"] != triage["gold"]][["ticket_id", "category", "gold"]]
if misses.empty:
    print("On this run, every prediction matched the labels.")
else:
    for _, m in misses.iterrows():
        text = tickets.loc[tickets["ticket_id"] == m["ticket_id"], "text"].iloc[0]
        print(f"{m['ticket_id']}: predicted {m['category']}, label {m['gold']}")
        print(f"   {text}\n")

On this run, every prediction matched the labels.

Where they disagree, the disagreement usually falls on one of these two-sided tickets; a clear blunder is the rarer case. This ambiguity is a property of the task that engineering cannot remove, and we address it directly in Chapter 9 with ways to measure agreement when reasonable people disagree and to score answers that have no single correct key.

3.7.1 Improving accuracy with examples

Debatable labels suggest their own remedy: if the boundary between two categories is a judgment call, show the model where we draw the line. A few-shot prompt puts labeled examples in the system message, and the natural examples to choose are the boundary cases. The baseline it improves on is the zero-shot prompt we have used so far, which gives the model only the task description, with no examples. We take three tickets, including the two ambiguous ones above, pair them with their gold labels, hold them out of scoring, and re-run the structured triage on the remaining twenty-seven, both ways.

# Three example tickets join the prompt; the other 27 stay as the test set.
example_ids = ["T013", "T021", "T024"]
examples = tickets[tickets["ticket_id"].isin(example_ids)]

example_block = "\n\n".join(
    f"Ticket: {row['text']}\nCategory: {gold[row['ticket_id']]}"
    for _, row in examples.iterrows()
)
fewshot_system = (
    "Classify the support ticket. Choose the single best category.\n\n"
    "Here are three labeled examples of how we categorize:\n\n" + example_block
)

held_out = tickets[~tickets["ticket_id"].isin(example_ids)]


def fewshot_triage(text: str) -> str:
    res = call_structured(prompt=text, schema=TicketAnalysis, system=fewshot_system)
    return res.data.category.value


with ThreadPoolExecutor(max_workers=8) as pool:
    fewshot_preds = list(pool.map(fewshot_triage, held_out["text"]))

fewshot_gold = held_out["ticket_id"].map(gold).tolist()
fewshot_acc = sum(p == g for p, g in zip(fewshot_preds, fewshot_gold)) / len(fewshot_preds)

# Zero-shot accuracy on the SAME 27 tickets, from the run already scored above.
zs = triage[~triage["ticket_id"].isin(example_ids)]
zeroshot_acc = (zs["category"] == zs["gold"]).mean()

print(f"zero-shot accuracy, 27 held-out tickets: {zeroshot_acc:.1%}")
print(f"few-shot accuracy, same 27 tickets:      {fewshot_acc:.1%}")

zero-shot accuracy, 27 held-out tickets: 100.0%
few-shot accuracy, same 27 tickets:      100.0%

Read this two ways. On the numbers: with twenty-seven tickets, a single ticket moves accuracy by nearly four points, so treat any gap here as directional, the way Chapter 9 will teach us to. On the mechanics: the example block is placed in the system message, which is precisely the stable prefix from Chapter 2. The examples repeat verbatim on every call, so once the prefix crosses the provider’s caching minimum, those extra example tokens are billed at the cached discount, and few-shot prompting becomes one of the cheapest accuracy improvements available: the full price is paid once, then about a tenth of the price on every later call (the cached-token rate the providers publish).¹

3.8 Validation as the first line of evaluation

This chapter carries a smaller lesson as well: the schema is itself a form of evaluation. Before we score anything against labels, validation has already rejected any reply that is not structured as a TicketAnalysis. A whole class of failures, a missing field, a category that does not exist, a priority that is not one of the three, can never reach our dataframe, because Instructor caught them and reasked. Typed output moves the discovery of such errors from months after deployment to the moment of the call. This check is the cheapest evaluation available, and it runs on every call without further effort.

Cost: this chapter’s example

Structured triage cost slightly more per ticket than the bare classification in Chapter 2, because the model also writes a reason and emits JSON. It remains a tiny fraction of a cent per ticket, which projects to well under a dollar for a full quarter. The choice of model still dominates the cost, and whether the output is structured barely registers. The same holds on self-hosted hardware, where the occasional validation reask adds no line to a bill and appears as added latency and occupied capacity.

Compliance: customer text in prompts

Every ticket we structured in this chapter is customer personal data leaving our environment in a prompt. Appendix D covers when that crosses into regulated territory and how to redact before the call.

3.9 Exercises

3.9.1 Conceptual questions

The Chapter 2 parser turned any unexpected reply into the category other. Why is that worse than raising an error?
1. The fallback adds a parsing step that slows down every call
2. It makes the category distribution look more uniform than it really is
3. It disguises a mistake as valid data, so the corruption goes unnoticed
4. It wastes tokens by sending each off-list reply back to the model
What does Pydantic validation give you that an English instruction in the prompt does not?
1. A shorter prompt, since the schema replaces the written instructions
2. A more careful model, because it can read the types it must satisfy
3. A cheaper call overall, since the validation step runs locally for free
4. A check enforced in code that the reply has the structure you specified
Instructor sometimes calls the model more than once for a single result. When?
1. When a reply fails validation: it sends the error back and reasks
2. On every call, because a second pass is needed to double-check the first
3. Whenever the temperature is above zero and the replies vary
4. Never; one call is always enough once a schema is attached
Two reasonable people label the same ticket differently. What does this imply for accuracy against a single label set?
1. The model is underperforming and needs a more detailed prompt
2. Accuracy has a ceiling below 100 percent on tickets like these
3. The labels are unreliable and should be redone before scoring
4. Accuracy should be swapped for a metric that skips hard tickets
You make a schema field required even though the value is only sometimes present in the source. What is the risk?
1. Validation rejects every document missing the field, halting the batch
2. Instructor reasks on each miss, multiplying the cost of the run
3. The field silently defaults to “other”, as in the loose parser
4. The model may invent a value just to satisfy the required field
In this chapter’s evaluation, what serves as the baseline?
1. The model’s published accuracy on a public classification benchmark
2. A classifier that picks at random among the fifteen categories
3. Chapter 2’s loose parser, scored on the same thirty tickets
4. A second human annotator labeling the same thirty tickets
Why does the chapter call schema validation a first line of evaluation?
1. It blocks malformed replies on every call, before any scoring
2. It maintains a running accuracy estimate as each call comes back
3. It compares every reply against the gold label automatically
4. It records each reply so failures can be audited months afterward
The chapter says the main benefit of structured output is not a higher accuracy score on clean inputs. What is it?
1. A lower per-call cost, because JSON replies are shorter than free text
2. Replies become typed objects or loud errors, never quiet corruption
3. Faster responses, because the model no longer writes explanations
4. Deterministic output, because a schema pins the temperature to zero

3.9.2 Build lab

Add a sentiment field to TicketAnalysis, constrained to "angry", "neutral", or "happy". Re-run the triage and show how priority and sentiment relate: are angry tickets mostly high priority? You may use an AI coding tool to draft the field. Read what it generates, and decide yourself whether sentiment should be required or optional.

3.9.3 Evaluate lab

Build a confusion matrix for the structured triage: for each gold category, which categories did the model predict? Use it to answer one question with a number: which category is the model worst at, and is that because the model is wrong or because the label is debatable? Choose how to present the matrix; defend which cell you call the “worst”.

Where we go next

Structured output is the backbone of everything practical we build later: it is how a RAG (Retrieval-Augmented Generation) system returns sources we can check (Chapter 7), how an agent decides which tool to call (Chapter 12), and how natural language becomes a SQL (Structured Query Language) query we can run (Chapter 14). In Chapter 4, however, we step off the API for a moment to look at where these models actually run and what changes when you host one yourself.

OpenAI and Anthropic both bill cached input tokens at roughly 0.1x the normal rate; see their pricing pages.↩︎

--- title: "Structured outputs" subtitle: "From prose to JSON to pandas" jupyter: gaba-core --- Our triage system from Chapter 2 has a hidden flaw: it asks the model for a category, and if it does not get a single clean word, it falls back to `other`, so errors that should be reported are silently recorded as valid data. In this chapter, we fix this by having the model return data that conforms to a Pydantic model that describes the exact structure of the output we want. Once the output is typed, we can load it directly into a dataframe and finally score the system against the labels for the first accuracy measurement of our triage. ::: {.callout-note title="Setup for this chapter"} Run in the `gaba-core` environment. This chapter introduces `gaba.llm.call_structured`, which uses the Instructor library to turn a model's reply into a validated Pydantic object. It is already installed in `gaba-core`. ::: ```{python} from dotenv import load_dotenv load_dotenv() import pandas as pd ``` ## Tools in this chapter | Tool | Why we use it here | Alternatives | Trade-off | |---|---|---|---| | **Pydantic** | describes the output structure as a typed class and validates data against it | Python dataclasses, jsonschema, hand-written checks | enforced types against a little boilerplate | | **Instructor** | wraps the model call to return a validated Pydantic object, retrying when validation fails | the providers' native structured-output modes | one behavior across every provider against an occasional retry cost | The trade-off against native structured outputs is weighed later in this chapter, and the [tooling-landscape appendix](../../appendices/i-tooling.qmd) lists the options. ## Where loose parsing breaks Recall the parser from Chapter 2: take the model's text, strip it, lowercase it, and if the result is not in our category list, call it `other`. That works only as long as the model returns exactly one bare category and nothing else. The moment we ask for anything more, it breaks down. Here is the result of asking for a category and a short reason in one free-text call, which is a completely reasonable thing to want: ```{python} from gaba.llm import call_llm from gaba.data import TICKET_CATEGORIES ticket = "The blender arrived shattered, glass everywhere. I want my money back." # Ask for a category and a reason together, as free text. loose = call_llm( prompt=ticket, system=( "Classify this support ticket. Reply in the form: category - reason. " "Choose the category from: " + ", ".join(TICKET_CATEGORIES) ), ) print("model said:", loose.text) # Now apply Chapter 2's parser to pull out the category. parsed = loose.text.strip().lower() category = parsed if parsed in TICKET_CATEGORIES else "other" print("parsed category:", category) ``` The model answered sensibly, yet the parser failed: the reply was a whole sentence where the parser expected a bare category, so it did not match the list and silently became `other`. A real category was lost without any error being raised. Run this across ten thousand tickets and you would have a column full of `other` that resembles data but is in fact corruption. The fragility extends beyond this naive parser to the format of the answer itself. Ask for several fields at once and the model obliges, but in whatever format it prefers. ```{python} from gaba.data import load_tickets ask = ("Give the category (one of: " + ", ".join(TICKET_CATEGORIES) + "), " "the priority (low, medium, or high), and a one-line reason.") for text in load_tickets()["text"].head(3): print(call_llm(prompt=text, system=ask).text.strip()) print("-" * 8) ``` Each answer is sensible, but the model chooses the format, and rarely the same format twice: a comma-separated line here, the dash-separated one above. To turn either into a dataframe you split on whichever separator the model happened to use, then hope the reason never contains one, and that the next model or the next prompt tweak does not switch the format again. The fields are present, but extracting them reliably, in the same way on every reply, is the fragile step. The deeper problem is that the contract between us and the model existed only as an English sentence in the prompt ("reply in the form..."), and English instructions are not enforced. Because an assumed structure carries no enforcement, the structure of the answer must be something the code checks. ## Describing the structure we want: Pydantic Pydantic lets us write that structure down as a class. Each field has a type, and Pydantic validates any data we put into it against those types. If the data does not fit, it raises an error, which keeps bad values from passing through. Here is the structure we actually want from a ticket: a category drawn from our fixed list, a priority that is one of three levels, and a one-sentence reason. ```{python} from enum import Enum from typing import Literal from pydantic import BaseModel, Field # Build an Enum from our category list so the schema and the data agree on # exactly which categories exist. The model will be allowed to choose only # these values, nothing else. TicketCategory = Enum("TicketCategory", {c: c for c in TICKET_CATEGORIES}, type=str) class TicketAnalysis(BaseModel): """The structured result we want for every ticket.""" category: TicketCategory priority: Literal["low", "medium", "high"] reason: str = Field(description="one short sentence explaining the choice") ``` The benefit is that invalid data cannot pass as valid. If something tries to set a category that is not on the list, Pydantic refuses it. ```{python} from pydantic import ValidationError # A good value validates fine. ok = TicketAnalysis(category="damaged_item", priority="high", reason="Glass arrived broken.") print("valid:", ok.category.value, ok.priority) # A bad category is rejected, loudly, instead of slipping through as "other". try: TicketAnalysis(category="please_help_me", priority="high", reason="x") except ValidationError as e: print("\nrejected, as it should be:") print(e.errors()[0]["msg"]) ``` The loud rejection is the point: where the loose parser hid the mistake, the validator reports it. The remaining step is getting the model to produce data in this structure in the first place. ## Typed output from a model The model's output has to leave the model and enter our code, and code needs structure it can rely on. JSON is the common format for that, a text format of keys and values that every programming language parses without extra work. The alternative is to let the model reply in sentences and parse them ourselves with regular expressions or string-splitting, which works until the phrasing shifts and then fails quietly, the flaw the previous section showed. JSON fixes the format of the output but leaves its meaning unchecked: it ensures the fields are present and typed, which is why we still validate the values. Other structured formats exist, such as XML (Extensible Markup Language) and YAML (a human-readable data format), but JSON is what the model providers support directly, so it is what we use. Modern models can return JSON on request, and several libraries make them return JSON that matches a Pydantic schema specifically. We use **Instructor**. It sends the model our schema, asks for a JSON object that fits, and validates the reply against the Pydantic model. If validation fails, it sends the error back to the model and asks again. What we get back is a validated `TicketAnalysis` with no string left for us to parse. We wrap this in `gaba.llm.call_structured`, which is like `call_llm` but takes a schema and returns the validated object as well as the usual cost bookkeeping. ::: {.callout-tip title="With an AI coding tool"} Drafting a Pydantic schema from an example of the JSON you want is a good use of an AI coding tool. Paste a sample object, ask for the model, then read every field: is the type right, is it required or optional, is the set of allowed values complete? The reading is the part that matters. ::: ```{python} from gaba.llm import call_structured result = call_structured(ticket, TicketAnalysis) analysis = result.data # a validated TicketAnalysis instance print("category:", analysis.category.value) print("priority:", analysis.priority) print("reason: ", analysis.reason) print(f"\ncost: ${result.cost_usd:.6f}") ``` The same ticket that broke the loose parser now returns as a typed object with `category` guaranteed to be one of our fifteen, `priority` guaranteed to be one of three, and a `reason` that we can read or ignore. There is no parsing step left to fail. ::: {.callout-warning title="Don't outsource this"} A tool can generate the schema, but it cannot decide for your business which fields are required and which are optional, or what should happen if a real document does not have a field you specified. Forcing the model to fill a field that is not in the source is how you manufacture hallucinations. These are your decisions, and we come back to them when extracting from real filings in Chapter 5. ::: ### Native structured outputs Instructor's validate-and-retry loop is not the only way to get schema-conforming output. The major providers have all offered native structured outputs since 2024-25: you pass a JSON schema in the request (`response_format` with a `json_schema` type in strict mode in the OpenAI protocol, or `responseSchema` in Google's Gemini), and the provider constrains the model's decoding with a grammar built from the schema. The model cannot emit a token that would break the JSON structure, so the output is guaranteed-valid JSON on the first try, with no retry loop and no reask cost. This book uses Instructor for two reasons. First, for consistency: we route every call through OpenRouter to reach many different models, and native strict mode is implemented unevenly across these models, while Instructor's reask loop works the same way everywhere. Second, for semantics: constrained decoding guarantees format while leaving meaning unchecked. A grammar can force `amount` to be a number, although it cannot check that the amount is positive, that a date is not in the future, or any other business rule. Pydantic validators can do that, and Instructor feeds their failures back to the model. The two approaches compose: format from the grammar, meaning from the validators. If you are committed to a single provider, use its native structured outputs as the base layer. It is faster, cheaper, and strictly more reliable at producing parseable JSON. Add semantic validation on top only where your fields have rules that a grammar cannot express. ### Validators carry the business rules In a refund pipeline, the simplest rule a grammar cannot express is that a refund must be a positive dollar amount. A schema can force `refund_usd` to be a number, but only a validator can insist on its sign. In Pydantic, that rule is a `field_validator` that rejects bad data no matter where it came from. ```{python} from pydantic import field_validator class RefundDecision(BaseModel): """A refund extracted from a ticket, with one business rule attached.""" refund_usd: float = Field(description="the refund amount in USD") @field_validator("refund_usd") @classmethod def must_be_positive(cls, v: float) -> float: if v <= 0: raise ValueError("refund_usd must be a positive dollar amount") return v # The rule fires on any invalid payload, model-made or hand-made. try: RefundDecision(refund_usd=-20.0) except ValidationError as e: print("rejected:", e.errors()[0]["msg"]) ``` Because Instructor feeds validation errors back to the model, this rule does more than reject: it triggers a reask, at our expense. Observing this requires a ticket designed to provoke the failure, one whose literal text contains a negative amount, and a way to count attempts. Instructor exposes hooks that fire on every raw completion, so counting completions counts attempts. ```{python} from gaba.llm import get_structured_client, cost_estimate, MODEL_DEFAULT # Each completion fires the hook once, so the list length is the attempt count # and each entry is what that attempt cost. attempt_costs = [] def record_attempt(response) -> None: attempt_costs.append(cost_estimate( MODEL_DEFAULT, response.usage.prompt_tokens, response.usage.completion_tokens)) client = get_structured_client() client.on("completion:response", record_attempt) # The trap: a faithful extraction of this ticket is a negative number, # which the validator will refuse. trap_ticket = ("Your agent promised me a refund of -$20 on yesterday's call. " "Please process it.") res = call_structured( prompt=trap_ticket, schema=RefundDecision, system="Extract the refund amount the customer says they were promised.", max_retries=3, ) client.off("completion:response", record_attempt) print(f"validated refund_usd: {res.data.refund_usd}") print(f"attempts: {len(attempt_costs)}") print(f"first attempt cost: ${attempt_costs[0]:.6f}; " f"reask cost: ${sum(attempt_costs[1:]):.6f}") ``` The object that comes back satisfies the rule either way; what varies is the path. When the model's first reply faithfully copies the negative number, the validator rejects it, Instructor sends the error back, and the attempt count above reads two or more, with the reask cost as the price of the correction. When the model silently fixes the sign on its own, the count reads one and the reask cost is zero. In our experience either outcome occurs depending on the model and the phrasing, which is exactly what the validator is for: it is the difference between hoping the model corrects bad data and guaranteeing that bad data cannot pass. ## Triage, now typed With `call_structured` and the schema, the triage loop becomes both simpler and safer. Every result is a typed object, so we read `.category` and `.priority` as attributes, with no text left to parse. ```{python} from gaba.data import load_tickets tickets = load_tickets() records = [] for _, row in tickets.iterrows(): res = call_structured( prompt=row["text"], schema=TicketAnalysis, system="Classify the support ticket. Choose the single best category.", ) records.append( { "ticket_id": row["ticket_id"], "category": res.data.category.value, "priority": res.data.priority, "cost_usd": res.cost_usd, } ) triage = pd.DataFrame(records) triage.head(8) ``` Every row is clean: no fallbacks, no off-list values, nothing silently dropped. The `category` column is drawn from exactly the fifteen we defined, and we obtained a `priority` in the same call, with no second request. ## When the schema demands what the source does not contain The schema is a contract that binds both parties: mark a field required and the model must fill it on every ticket, including the tickets where the value simply is not there. The most common real extraction failure we know of is a schema that forces the model to answer a question the document never answers, a failure easy to misattribute to the model misreading the document. The "Don't outsource this" warning above said the required-versus-optional choice is yours; here is what getting it wrong looks like. Ten fictional tickets, of which exactly three mention an order ID. The gold list is written by hand: an ID where one exists, `None` where none does. ```{python} # Only the first three tickets contain an order ID. id_tickets = [ "Order ORD-58213 arrived with the wrong color headphones.", "Still waiting on ORD-91447, it has been two weeks now.", "I was double charged for order ORD-20786 and want one charge reversed.", "The app crashes every time I open the settings page.", "Your driver left the package in the rain and everything inside is soaked.", "How do I change the email address on my account?", "The subscription renewed even though I cancelled last month.", "The blender is much louder than the demo video suggested.", "I never received the discount code your newsletter promised.", "Two of the mugs in my delivery were chipped at the rim.", ] gold_ids = ["ORD-58213", "ORD-91447", "ORD-20786", None, None, None, None, None, None, None] ``` Two schemas, identical except that the second allows `order_id` to be `None`. ```{python} class OrderRefRequired(BaseModel): order_id: str = Field(description="the order ID referenced in the ticket") class OrderRefOptional(BaseModel): order_id: str | None = Field( None, description="the order ID referenced in the ticket, or null if the ticket gives none" ) ``` Now both schemas extract from all ten tickets. Twenty calls, run in parallel. ```{python} from concurrent.futures import ThreadPoolExecutor def extract_id(ticket: str, schema: type[BaseModel]): # A required field has a third possible outcome besides right and wrong: # if the model never satisfies the schema, Instructor gives up and raises. # We record that outcome instead of letting it stop the batch. try: res = call_structured(prompt=ticket, schema=schema, system="Extract the order ID from this support ticket.") return res.data.order_id except Exception: return "<failed validation>" with ThreadPoolExecutor(max_workers=8) as pool: required_ids = list(pool.map(lambda t: extract_id(t, OrderRefRequired), id_tickets)) optional_ids = list(pool.map(lambda t: extract_id(t, OrderRefOptional), id_tickets)) comparison = pd.DataFrame({ "ticket": [t[:44] + ("..." if len(t) > 44 else "") for t in id_tickets], "gold": gold_ids, "required_schema": required_ids, "optional_schema": optional_ids, }) comparison ``` ```{python} # Score only the seven tickets whose gold answer is "there is no ID". no_id_rows = [i for i, g in enumerate(gold_ids) if g is None] filled_anyway = sum(1 for i in no_id_rows if required_ids[i] and required_ids[i] != "<failed validation>") errored_out = sum(1 for i in no_id_rows if required_ids[i] == "<failed validation>") correct_none = sum(1 for i in no_id_rows if optional_ids[i] is None) print(f"tickets with no order ID: {len(no_id_rows)}") print(f"required schema returned a value anyway: {filled_anyway}/{len(no_id_rows)}") print(f"required schema failed outright: {errored_out}/{len(no_id_rows)}") print(f"optional schema correctly returned None: {correct_none}/{len(no_id_rows)}") ``` Read the `required_schema` column on the no-ID rows. The model has to put *something* there, so it reaches for whatever satisfies the contract: a placeholder, an empty-looking token, or, worst of all, an invented ID with exactly the right format. An invented `ORD-` number is indistinguishable from a real one downstream, which is what makes this failure so expensive in practice; the occasional call that fails outright after its retries is the least damaging outcome, because that failure at least is visible. The optional schema gives the model a truthful way out, and it mostly takes it. The resulting rule: a field is required only when its absence from the source would itself be an error; everything else is `str | None`, and the `None`s are data. We meet this rule again on real filings in Chapter 5. ## Evaluation: accuracy against gold labels The evaluation promised in Chapter 1 is now possible. In Chapter 2, we could report only cost, because we did not have any labels to check our answers against; we have those labels now. **Metric**: classification accuracy, the fraction of tickets whose predicted category matches the gold label.\ **Test set**: the thirty tickets, hand-labeled in `tickets_labels.csv`.\ **Baseline**: the loose free-text parser from Chapter 2. The question is whether structuring the output improved the accuracy or only the format. First, load the gold labels and score the structured triage. ```{python} from gaba.data import load_ticket_labels labels = load_ticket_labels() gold = dict(zip(labels["ticket_id"], labels["category"])) triage["gold"] = triage["ticket_id"].map(gold) structured_acc = (triage["category"] == triage["gold"]).mean() print(f"structured accuracy: {structured_acc:.1%} " f"({(triage['category'] == triage['gold']).sum()}/{len(triage)})") ``` This score also revisits the comparison opened in Chapter 1, where the era-one keyword rules reached 73 percent on these same thirty tickets against these same labels, and their misses remain fixed until someone rewrites the rules by hand. This gap is the intent ladder of Chapter 1 made measurable, since the rules match exact words, while the generative system responds to what the customer meant, whatever words carried it, and it did so without knowing the queue's vocabulary in advance. For the baseline, we run the Chapter 2 loose parser over the same tickets and score it the same way, and we also count how many tickets it silently sent to "other". ```{python} def loose_triage(text: str) -> str: """Chapter 2's parser: free-text category, fall back to 'other'.""" r = call_llm( prompt=text, system=( "Reply with exactly one category, lowercase, nothing else: " + ", ".join(TICKET_CATEGORIES) ), ) answer = r.text.strip().lower() return answer if answer in TICKET_CATEGORIES else "other" loose_preds = [loose_triage(t) for t in tickets["text"]] loose_acc = sum(p == g for p, g in zip(loose_preds, triage["gold"])) / len(loose_preds) fell_to_other = sum(p == "other" for p in loose_preds) print(f"loose-parser accuracy: {loose_acc:.1%}") print(f"loose parser sent {fell_to_other} of {len(loose_preds)} tickets to 'other'") print(f"structured accuracy: {structured_acc:.1%}") ``` On accuracy the two paths are nearly identical, and on these clean tickets the loose parser never even fell back to `other`. Although structuring did not make the model a better classifier, its effect appears in a property that accuracy cannot measure. Ask the model to name each ticket's issue in its own words, with no fixed list, and the effect becomes visible. ```{python} from concurrent.futures import ThreadPoolExecutor def freeform_label(text: str) -> str: # No list, no schema: the model phrases each ticket's issue however it likes. r = call_llm(prompt=text, system="In two to four words, name the customer's main issue. " "Reply with only the phrase, lowercase, no punctuation.") return r.text.strip().lower().rstrip(".") with ThreadPoolExecutor(max_workers=8) as pool: freeform = list(pool.map(freeform_label, tickets["text"])) print(f"free-form labels: {len(set(freeform))} distinct for {len(freeform)} tickets") print(f"schema categories: {triage['category'].nunique()} distinct for {len(triage)} tickets") ``` ```{python} #| label: fig-label-fragmentation #| fig-cap: "Distinct labels produced for the same thirty tickets. Asked for a free-text label, the model returns nearly one unique string per ticket, so nothing can be grouped or counted; the schema-constrained run collapses them to the bounded set of categories it is allowed to use." #| fig-alt: "Bar chart comparing distinct label counts for thirty tickets: a tall gray bar for free-form labels near thirty, and a much shorter blue bar for schema categories." import matplotlib.pyplot as plt counts = [len(set(freeform)), triage["category"].nunique()] fig, ax = plt.subplots(figsize=(5.2, 3.3)) bars = ax.bar(["free-form label", "schema category"], counts, color=["#9a9a9a", "#0969da"], width=0.55) ax.bar_label(bars, padding=3) ax.set_ylabel("distinct labels for 30 tickets") ax.set_ylim(0, len(freeform) + 4) ax.spines[["top", "right"]].set_visible(False) plt.tight_layout() plt.show() ``` The result is nearly one distinct string per ticket. The same shipping delay comes back as `late delivery`, then `delayed package`, then `still waiting on my order`, each a different string, and strings that cannot be grouped cannot be counted: ask how many shipping complaints arrived this week and the free-form column has no answer. The schema-constrained run is bounded by design, every value one of the categories we defined, so the thirty tickets collapse into a countable handful that totals, filters, and joins like any other column. Although structured output did not raise the accuracy here, it made the output usable as data. This usability is the primary argument for a schema, and it compounds with every field added: one free-text field fragments, three fields fragment three ways, and a parser written for the model's current format breaks on its next. ```{mermaid} %%| echo: false %%| label: fig-loose-vs-structured %%| fig-cap: "The same model reply down two paths. The loose parser turns any unexpected reply into 'other' and lets it into the dataframe, which is silent corruption; schema validation returns either a typed object or a loud error that Instructor can retry, never a wrong answer that looks like a right one." flowchart TB subgraph loose["loose path: fails silently"] reply1["model reply (free text)"] --> parse["strip, lowercase, match against the list"] parse --> corrupt["off-list reply becomes 'other' and enters the dataframe"] end subgraph typed["structured path: fails loudly"] reply2["model reply (JSON)"] --> valid["Pydantic schema validation"] valid -- "fits" --> obj["typed TicketAnalysis"] valid -- "does not fit" --> err["loud ValidationError; Instructor reasks the model"] end loose ~~~ typed ``` A high accuracy score also does not mean the task is unambiguous. Consider two tickets in our set. "Take me off my plan immediately and stop charging my card" sits between a cancellation and a subscription change. "I paid for express shipping and it arrived later... I want the shipping fee back" sits between a refund and a billing problem. Whether the model happened to match our label on these or not, the labels themselves are debatable, which puts a ceiling below 100 percent on accuracy against any single label set. We can list which tickets, if any, the model and our labels disagreed on: ```{python} misses = triage[triage["category"] != triage["gold"]][["ticket_id", "category", "gold"]] if misses.empty: print("On this run, every prediction matched the labels.") else: for _, m in misses.iterrows(): text = tickets.loc[tickets["ticket_id"] == m["ticket_id"], "text"].iloc[0] print(f"{m['ticket_id']}: predicted {m['category']}, label {m['gold']}") print(f" {text}\n") ``` Where they disagree, the disagreement usually falls on one of these two-sided tickets; a clear blunder is the rarer case. This ambiguity is a property of the task that engineering cannot remove, and we address it directly in Chapter 9 with ways to measure agreement when reasonable people disagree and to score answers that have no single correct key. ### Improving accuracy with examples Debatable labels suggest their own remedy: if the boundary between two categories is a judgment call, show the model where we draw the line. A **few-shot** prompt puts labeled examples in the system message, and the natural examples to choose are the boundary cases. The baseline it improves on is the **zero-shot** prompt we have used so far, which gives the model only the task description, with no examples. We take three tickets, including the two ambiguous ones above, pair them with their gold labels, hold them out of scoring, and re-run the structured triage on the remaining twenty-seven, both ways. ```{python} # Three example tickets join the prompt; the other 27 stay as the test set. example_ids = ["T013", "T021", "T024"] examples = tickets[tickets["ticket_id"].isin(example_ids)] example_block = "\n\n".join( f"Ticket: {row['text']}\nCategory: {gold[row['ticket_id']]}" for _, row in examples.iterrows() ) fewshot_system = ( "Classify the support ticket. Choose the single best category.\n\n" "Here are three labeled examples of how we categorize:\n\n" + example_block ) held_out = tickets[~tickets["ticket_id"].isin(example_ids)] def fewshot_triage(text: str) -> str: res = call_structured(prompt=text, schema=TicketAnalysis, system=fewshot_system) return res.data.category.value with ThreadPoolExecutor(max_workers=8) as pool: fewshot_preds = list(pool.map(fewshot_triage, held_out["text"])) fewshot_gold = held_out["ticket_id"].map(gold).tolist() fewshot_acc = sum(p == g for p, g in zip(fewshot_preds, fewshot_gold)) / len(fewshot_preds) # Zero-shot accuracy on the SAME 27 tickets, from the run already scored above. zs = triage[~triage["ticket_id"].isin(example_ids)] zeroshot_acc = (zs["category"] == zs["gold"]).mean() print(f"zero-shot accuracy, 27 held-out tickets: {zeroshot_acc:.1%}") print(f"few-shot accuracy, same 27 tickets: {fewshot_acc:.1%}") ``` Read this two ways. On the numbers: with twenty-seven tickets, a single ticket moves accuracy by nearly four points, so treat any gap here as directional, the way Chapter 9 will teach us to. On the mechanics: the example block is placed in the system message, which is precisely the stable prefix from Chapter 2. The examples repeat verbatim on every call, so once the prefix crosses the provider's caching minimum, those extra example tokens are billed at the cached discount, and few-shot prompting becomes one of the cheapest accuracy improvements available: the full price is paid once, then about a tenth of the price on every later call (the cached-token rate the providers publish).^[OpenAI and Anthropic both bill cached input tokens at roughly 0.1x the normal rate; see their pricing pages.] ## Validation as the first line of evaluation This chapter carries a smaller lesson as well: the schema is itself a form of evaluation. Before we score anything against labels, validation has already rejected any reply that is not structured as a `TicketAnalysis`. A whole class of failures, a missing field, a category that does not exist, a priority that is not one of the three, can never reach our dataframe, because Instructor caught them and reasked. Typed output moves the discovery of such errors from months after deployment to the moment of the call. This check is the cheapest evaluation available, and it runs on every call without further effort. ::: {.callout-tip title="Cost: this chapter's example"} Structured triage cost slightly more per ticket than the bare classification in Chapter 2, because the model also writes a reason and emits JSON. It remains a tiny fraction of a cent per ticket, which projects to well under a dollar for a full quarter. The choice of model still dominates the cost, and whether the output is structured barely registers. The same holds on self-hosted hardware, where the occasional validation reask adds no line to a bill and appears as added latency and occupied capacity. ::: ::: {.callout-important title="Compliance: customer text in prompts"} Every ticket we structured in this chapter is customer personal data leaving our environment in a prompt. Appendix D covers when that crosses into regulated territory and how to redact before the call. ::: ## Exercises ### Conceptual questions 1. The Chapter 2 parser turned any unexpected reply into the category `other`. Why is that worse than raising an error? A. The fallback adds a parsing step that slows down every call B. It makes the category distribution look more uniform than it really is C. It disguises a mistake as valid data, so the corruption goes unnoticed D. It wastes tokens by sending each off-list reply back to the model 2. What does Pydantic validation give you that an English instruction in the prompt does not? A. A shorter prompt, since the schema replaces the written instructions B. A more careful model, because it can read the types it must satisfy C. A cheaper call overall, since the validation step runs locally for free D. A check enforced in code that the reply has the structure you specified 3. Instructor sometimes calls the model more than once for a single result. When? A. When a reply fails validation: it sends the error back and reasks B. On every call, because a second pass is needed to double-check the first C. Whenever the temperature is above zero and the replies vary D. Never; one call is always enough once a schema is attached 4. Two reasonable people label the same ticket differently. What does this imply for accuracy against a single label set? A. The model is underperforming and needs a more detailed prompt B. Accuracy has a ceiling below 100 percent on tickets like these C. The labels are unreliable and should be redone before scoring D. Accuracy should be swapped for a metric that skips hard tickets 5. You make a schema field required even though the value is only sometimes present in the source. What is the risk? A. Validation rejects every document missing the field, halting the batch B. Instructor reasks on each miss, multiplying the cost of the run C. The field silently defaults to "other", as in the loose parser D. The model may invent a value just to satisfy the required field 6. In this chapter's evaluation, what serves as the baseline? A. The model's published accuracy on a public classification benchmark B. A classifier that picks at random among the fifteen categories C. Chapter 2's loose parser, scored on the same thirty tickets D. A second human annotator labeling the same thirty tickets 7. Why does the chapter call schema validation a first line of evaluation? A. It blocks malformed replies on every call, before any scoring B. It maintains a running accuracy estimate as each call comes back C. It compares every reply against the gold label automatically D. It records each reply so failures can be audited months afterward 8. The chapter says the main benefit of structured output is not a higher accuracy score on clean inputs. What is it? A. A lower per-call cost, because JSON replies are shorter than free text B. Replies become typed objects or loud errors, never quiet corruption C. Faster responses, because the model no longer writes explanations D. Deterministic output, because a schema pins the temperature to zero ### Build lab Add a `sentiment` field to `TicketAnalysis`, constrained to `"angry"`, `"neutral"`, or `"happy"`. Re-run the triage and show how priority and sentiment relate: are angry tickets mostly high priority? You may use an AI coding tool to draft the field. Read what it generates, and decide yourself whether sentiment should be required or optional. ### Evaluate lab Build a confusion matrix for the structured triage: for each gold category, which categories did the model predict? Use it to answer one question with a number: which category is the model worst at, and is that because the model is wrong or because the label is debatable? Choose how to present the matrix; defend which cell you call the "worst". ::: {.callout-note title="Where we go next"} Structured output is the backbone of everything practical we build later: it is how a RAG (Retrieval-Augmented Generation) system returns sources we can check (Chapter 7), how an agent decides which tool to call (Chapter 12), and how natural language becomes a SQL (Structured Query Language) query we can run (Chapter 14). In Chapter 4, however, we step off the API for a moment to look at where these models actually run and what changes when you host one yourself. :::