1  Three eras of analytics

From rules and ML to generative systems

Consider a Monday morning on the analytics desk. The support manager forwards a folder of 10,000 customer support tickets from the last quarter and asks for three things: a breakdown of the categories the tickets fall into, a list of the top recurring complaints, and a rough sense of which tickets contain churn-risk language. She wants the answers by Friday, and she has asked some version of this question every quarter for the past five years.

The analytics techniques suited to this request were different five years ago, different again two years ago, and are different once more today. Each produces its own kind of useful outputs, demands its own kind of input-processing effort, and has its own unique failure modes. This chapter explains which technique fits which kind of problem, and the rest of the book teaches how to build and measure systems with the newest of these techniques.

NoteSetup for this chapter

The one code cell in this chapter runs locally in the gaba-core environment. If you have not set up this environment yet, Appendix A has complete setup instructions, from a clean machine to a verified installation.

1.1 Three eras of text analysis

What an analyst could realistically build in plain Python falls into three eras, each defined less by the tools at hand than by where the analyst’s effort concentrated.

Era one: rules. We begin by reading a few hundred tickets to learn the vocabulary of the queue, then encode what we find as regular expressions. Tickets that mention refund sort into one category, those that mention broken or doesn’t work into another, and a deliberate exception captures the ones that cite both a refund and a shipping delay, since those tend to be the angriest. By Thursday the system classifies perhaps seventy percent of the tickets correctly and misclassifies the rest. Its value is evident, because it is fast and auditable enough that the support manager can read the rules herself and confirm they make sense. Its weakness is equally genuine: the rules are brittle, and the first time a new product launches and the vocabulary shifts, they fall out of date within a week.

That era is computationally cheap enough to revisit right now. Here is a working era-one system for a thirty-ticket sample of the queue: keyword rules checked in order, first match wins, scored against hand-assigned gold categories. The data loaders come from the book’s shared gaba package, introduced properly in the next two chapters.

TipWith an AI coding tool

Drafting the first pass of a RULES list like the one below is a reasonable ask for an assistant: paste a dozen sample tickets and have it propose keyword-category pairs. The assistant cannot decide which misses are acceptable, so read its proposals against tickets you have actually read. A pair that maps “feature” to feature requests looks self-evident, yet it misroutes the customer whose export feature stopped working. Misclassifications like this are invisible in the aggregate accuracy reported below.

# An era-one triage system in miniature. Each rule is a (keyword, category)
# pair; the first keyword found in the ticket decides, and no match means "other".
from gaba.data import load_tickets, load_ticket_labels

RULES = [
    ("refund", "refund_request"), ("charg", "billing_problem"), ("invoice", "billing_problem"),
    ("cracked", "damaged_item"), ("defect", "damaged_item"), ("instead", "wrong_item"),
    ("someone else", "wrong_item"), ("deliver", "shipping_delay"), ("where is my", "shipping_delay"),
    ("password", "account_access"), ("locked out", "account_access"), ("cancel", "cancellation"),
    ("switch", "subscription_change"), ("upgrade", "subscription_change"),
    ("used my card", "fraud_report"), ("error", "technical_issue"),
    ("would be great", "feature_request"), ("feature", "feature_request"),
    ("worst", "complaint"), ("fantastic", "praise"), ("thank", "praise"),
]

def classify(text: str) -> str:
    for keyword, category in RULES:
        if keyword in text.lower():
            return category
    return "other"

tickets, labels = load_tickets(), load_ticket_labels()
gold = dict(zip(labels["ticket_id"], labels["category"]))
predictions = [classify(t) for t in tickets["text"]]
correct = sum(p == gold[t] for p, t in zip(predictions, tickets["ticket_id"]))
print(f"keyword rules: {correct}/{len(tickets)} tickets correct ({correct / len(tickets):.0%})")
keyword rules: 22/30 tickets correct (73%)

A simple rule-based system reaches seventy-three percent classification accuracy with no model training and no meaningful compute cost. The misclassifications show part of why such a system is brittle: the customer who writes “take me off my plan immediately and stop charging my card” is categorized as a billing problem, because a rule matched the word charging and the cancellation intent could not be captured by simple word-matching. This seventy-three percent serves as the era-one baseline: Chapter 3 evaluates a generative system on the same thirty tickets against the same labels.

Era two: traditional machine learning. We hand-label a thousand tickets across the categories that are important in our domain, represent each ticket as a TF-IDF vector (a weighted count of which words appear, with common words discounted) and train a classifier on those numbers, a logistic regression or a small gradient-boosted model, reaching perhaps ninety percent accuracy on a held-out test set. The result is far more robust to variation in phrasing, because a thousand labeled examples expose the classifier to many wordings of the same intent. The model, however, is opaque in a new way: the support manager can no longer read the reasoning behind any single classification, though she can read a confusion matrix (a table of predicted against actual categories) and trust the headline number. The labor has shifted from writing rules to labeling examples, and the risk has shifted with it, from missed edge cases to distribution shift. When next quarter’s tickets arrive from a subtly different distribution, a shift not immediately evident to the analyst or the manager, the model can degrade quietly, and no one notices until customers complain about misrouted tickets.

Era three: generative systems. We write a prompt that instructs a model to read each ticket and return a JSON object (a structured text format of keys and values that code can read directly) with three fields: category, chosen from a list of fifteen, top_complaint, and churn_risk. The prompt goes to a foundation model (a large neural network pretrained on broad text, adaptable to many tasks), whose training text includes the very phrases our customers use, and we run it across all ten thousand tickets without training anything ourselves. It fails and succeeds in unfamiliar ways: it makes mistakes the era-two model never would, such as occasionally returning the opposite of what we asked, yet it recognizes intent that no wordlist could reach, such as the customer who requests a refund without once using the word. The system is more flexible, more expensive per ticket, and opaque in yet a third way, because neither the prompt nor the model’s weights explain any single answer. The analyst’s effort therefore moves once more, into writing the prompt, choosing the model, and evaluating the output against examples we already trust.

Each era demands roughly the same amount of work from the analyst; what changes is the task that work goes into: writing rules in the first era, labeling examples in the second, and framing the problem and measuring the result in the third. The eras can also be read as successive steps from words toward intent: rules match exact words, a trained classifier matches patterns of wording, and a generative model responds to what the customer meant, irrespective of the words used by the customer.

Table 1.1: Three eras of text analysis, compared on what we build, where the analyst’s effort concentrates, and the failure mode to guard against.
Era one: rules Era two: traditional ML Era three: generative
What we build keyword rules encoding the queue’s vocabulary a classifier trained on labeled examples a prompt for a foundation model
Where effort concentrates writing and maintaining the rules labeling a thousand tickets well framing the problem, measuring the result
Failure mode to guard against rules go stale when the vocabulary shifts quiet degradation under distribution drift confident errors, if left unmeasured

1.2 What changed and what did not

Because the shift is easy both to overstate and to understate, it is worth being precise about what is new, what is not, and what sits in between.

1.2.1 What is genuinely new

Three things are new in a way that should change the systems we build.

Zero-shot capability that often works. A modern foundation model can do a task it was never explicitly trained on, in any of dozens of languages, on the strength of a paragraph of instructions. While this capability is neither magic nor reliable across the board, the tasks a business analyst is asked to do (classify this, extract that, summarize the other, route based on a rule) are precisely the ones where it works often enough to be worth our time.

Instruction following in plain English. Where the earlier eras required engineered features or hand-written rules, the instruction to a modern model is written in plain English, which the model maps to behavior. That sounds trivial, but it changes who can build what: a senior analyst with a few days of Python can now build systems that, in 2020, required a machine-learning engineer.

Tool use and structured outputs. A modern model can return JSON that conforms to a schema, call tools, plan multi-step actions, and feed the results back into its own next step. Structured output is the bridge from “the model said something” to “I can put it in a dataframe,” and tool use extends the same bridge to “the model did something.” We develop this capability in Chapter 3 (structured outputs) and put it into action in Chapters 11 (workflows) and 12 (agents).

1.2.2 What is not new

The big mistakes in this field come from forgetting what did not change.

We still have to evaluate. A model that says something plausibly correct may nonetheless be wrong. The single most common failure in generative AI projects is putting a system into use that nobody knows how to measure, discovering three months later that it has been quietly wrong, and not knowing when the errors began. This failure is not hypothetical: in May 2026, EY Canada withdrew a published cybersecurity report after an outside review found fabricated and misattributed citations behind most of its footnotes, along with internally inconsistent figures, in a document the firm had been putting in front of clients for months.1 Measuring what you build is part of building it; a common mistake is to treat measurement as a step bolted on at the end.

Bad data still poisons everything. If we ask the model to extract a field that is only sometimes present in the documents, the model will hallucinate that field for the documents where it is absent. The remedy predates generative models: a held-out test set, a baseline, and quantifiable numbers.

Probabilistic outputs are still probabilistic. The same prompt against the same model can produce different outputs depending on temperature (a setting that controls randomness), sampling, and the model provider’s internal state. A system designed as if these outputs were deterministic will fail when they vary, and a system designed as if they were reliably correct will pass the model’s errors through unchecked, with potential to propogate through automation pipelines with expensive consequences.

Cost matters and compounds. Unlike the first two eras, which were free at inference time, inference in the third era is metered: each classified ticket, summarized page, and agent step carries a per-call charge, and those charges scale with volume. Building a system that performs the task well is therefore only half the job, because the system must also pay for itself. The cost callout in every applied chapter works out the economics of the system built there.

1.2.3 What is maybe new

One thing has shifted enough that it deserves its own category: an analyst can now build systems that act, going beyond systems that report. A dashboard tells us what is happening, and a traditional ML model tells us what is likely to happen, while a generative system can, if we let it, read an email, look something up in our CRM (the customer-records system), draft a response, and, if we really let it, send the response. Because the risk profile changes once a system can take actions, safety and governance get their own chapters in Part IX. The capability still belongs among the options an analyst weighs, because some problems, such as routing a ticket onward once it has been categorized, are only fully served by a system that acts. Systems of this kind, called agentic systems, have become a central pattern in applied generative AI, and Part IV builds them.

1.3 The three eras side by side

Table 1.1 summarized the narrative of the three eras. Choosing between them in practice involves several further criteria, from cost at inference to auditability, some of which are shown in the following table.

Criterion Rules Traditional ML Generative systems
Cost at inference Free Almost free Pay per call
Effort to start Days Weeks (labeling) Hours
Effort to improve Linear (more rules) Linear (more labels) Sub-linear (better prompts, retrieval, eval)
Handles new vocabulary Poorly Moderately Well
Handles novel tasks No No (retrain) Often yes (zero-shot)
Auditability High (readable rules) Medium (confusion matrix) New techniques (Ch 9, Ch 24)
Determinism High Medium Low (without care)
Risk of confident error Low Medium High
Can take actions No No Yes (Ch 11, 12)

Every cell of this table is the topic of at least one later chapter. None of these criteria is settled in the field, but the working rule when in doubt is to build the simplest thing that meets the bar and then measure it.

Expressed in business terms, the same comparison takes a different form: the table below prices the 10,000-ticket quarterly job for each era. Although the numbers are illustrative, the trade-off structure is real, because each era exchanges fixed effort, marginal cost, and the cost of change in its own proportions.

The 10,000-ticket job, per era (illustrative) Rules Traditional ML Generative systems
Build effort ~4 analyst-days ~15 analyst-days, mostly labeling ~2 analyst-days, mostly evaluation
Marginal cost per quarter ~$0 ~$0 well under $1
Adapting to a new product line days of rewriting rules weeks of relabeling and retraining hours of editing the prompt, plus re-measuring

1.4 The model landscape

“Generative AI” covers several distinct families of model, organized most clearly by what each family produces and the inputs it accepts. The newest text models accept several of those inputs at once, text together with an image or audio in the same call, so a model is described by the set of inputs it accepts, which may contain more than one type. The table below lists every family this book builds with, tagged with its chapters, so it doubles as a map of what is ahead in the book.

Figure 1.1: The generative model landscape organized by what each family produces and the inputs it accepts. The newest text models take several inputs in a single call; the + marks them. Click a row for what the family does, representative open and hosted models with links to their model cards, and a business use; the last column shows where this book builds with each family.

The model names in the table carry a distinction worth understanding now. Pretraining produces a base model, a machine that continues whatever text it is given, fluently but unhelpfully, like a new hire who has read every book in the library but has never sat in a meeting. Ask a base model to write a poem and it may list poetry exercises, because that is what often follows such requests in its training data. Instruction tuning and preference training, the on-the-job training that turns continuation into answering, produce an instruct model; every large language model (LLM) this book uses is of this kind, and the suffix in names like Llama-3.1-8B-Instruct signifies exactly this step. Chapter 2 returns to the distinction with an interactive demo, right before we call a model for the first time.

flowchart LR
    pre["pretraining<br/>(predict the next token<br/>over the internet)"] --> base["base model:<br/>a text continuer"]
    base --> sft["supervised fine-tuning<br/>(instruction-response pairs)"]
    sft --> pref["preference tuning<br/>(which answer do people prefer?)"]
    pref --> inst["instruct model:<br/>the assistant you call"]
Figure 1.2: How a base model becomes the assistant you actually call: pretraining produces a text continuer; supervised fine-tuning on instruction-response pairs teaches the answering format; preference tuning teaches which answers people prefer. The same pipeline, run on your own examples, is Chapter 16’s fine-tuning in miniature.

One more distinction matters when you read a model card: most large models today are mixture-of-experts designs that advertise two parameter counts, total and active. Chapter 4 explains why both numbers matter in operational terms.

1.5 Where this book takes us

The rest of the book assembles, in order, the layers of one working system. From here forward we are on the analytics desk at a mid-sized company, and by the last chapter the system we build there does three jobs: it triages the incoming ticket queue, it answers questions over the company’s filings and reports, and it runs an agent when a question takes more than one step. Most chapters add a layer to this system, and a few add a capability the desk reaches for when the work calls for it, so the pieces accumulate the way they do in practice.

flowchart TB
    s1["Frame the question:<br/>is this even an LLM job? (Chapter 1)"]
    s2["Reach a model:<br/>calling it, structured output, where it runs (Part I)"]
    s3["Prepare the data:<br/>extract text from documents and embed it (Part II)"]
    s4["Retrieve and ground:<br/>find the few passages that answer it (Parts II and III)"]
    s5["Orchestrate:<br/>one call, a workflow, or an agent (Part IV)"]
    s6["Specialize as needed:<br/>analytics, a custom model, other modalities, knowledge graphs (Parts V to VIII)"]
    s7["Deploy and operate:<br/>measure it, watch it, and guard it (Part IX)"]
    s1 --> s2 --> s3 --> s4 --> s5 --> s6 --> s7
Figure 1.3: The journey of a business question, from deciding whether it is even an LLM job to a deployed, auditable answer. Each stage is a part of the book, and evaluation, cost, and compliance run under all of them. Later chapters point back to where they sit on this map.

We start with the substrate (Part I): how to call a model, get structured output, and pick where the model runs. Then we build the document layer (Parts II and III): how to extract text from PDFs, search over it with embeddings, build a basic retrieval-augmented system, and improve it with better retrieval techniques. In Part III we reach the evaluation toolkit (Chapter 9), after which everything is easier to judge, because we finally have a way to tell whether each thing we build is better than the last.

From there we move into workflows and agents (Part IV), embedding analytics and natural-language Q&A over structured data (Part V), customization of the underlying model (Part VI), modalities beyond text (Part VII), and structured knowledge (Part VIII). The last three chapters (Part IX) are about turning the working system we have built into something a stakeholder can actually rely on: safety, observability, and a small chat interface that sits on top of everything we have built. Part X closes with capstone projects, each of which demands several of the book’s techniques in combination, as real assignments do. For example, one of the projects is a responsible resume screener whose bias audit is part of the build.

If we only had time for three chapters, they would be Chapter 2 (calling models), Chapter 7 (the basic RAG system), and Chapter 9 (evaluation). Those three give us the smallest working system and the discipline to know if it is any good.

1.6 Matching a problem to a technique

Each era added a tool to the analyst’s repertoire without retiring the ones before it, so the practical question is which technique the problem’s requirements select, a question that the newness of any technique does not answer by itself. Where the logic is explicit and every decision must be auditable, rules remain the right choice, because anyone can read them and confirm what the system will do. Where labeled examples are plentiful, the vocabulary is stable, and the volume demands near-zero marginal cost, traditional ML offers high accuracy at almost no cost per item. Where the task is specified in written instructions, where phrasing varies freely, or where no training data exists, a generative system is the tool that applies; and where the problem requires taking actions beyond producing a report, an agentic system, introduced in Part IV, extends the generative approach with the ability to act. These considerations compress into four questions, asked in order:

flowchart TB
    d1{"Can the logic be written as<br/>explicit rules a colleague<br/>could follow?"} -- yes --> rulesleaf["rules"]
    d1 -- no --> d2{"Plentiful labeled examples<br/>and a stable vocabulary?"}
    d2 -- yes --> ml["traditional ML"]
    d2 -- no --> d3{"Near-zero marginal cost<br/>per item required?"}
    d3 -- "yes: invest in labels" --> ml
    d3 -- no --> d4{"Must the system take<br/>actions beyond reporting?"}
    d4 -- no --> gen["generative system"]
    d4 -- yes --> agentic["agentic system<br/>(Part IV)"]
Figure 1.4: Four questions that match a problem to a technique. Logic you can write down stays in rules; plentiful labels with a stable vocabulary point to traditional ML, as does a hard requirement for near-zero marginal cost; otherwise the problem is generative, and if the system must go beyond reporting and take actions, it is agentic.

A fair test before moving on is to take a problem your own team is facing and walk it through the four questions. If the walk stalls, the stall itself is informative, because each question names a fact about your situation, such as how stable the vocabulary is, how many labeled examples exist, and whether the volume demands near-zero marginal cost, and the fact you cannot supply is the one to go and establish. The next twenty-four chapters assume this placement judgment and build on it.

WarningDon’t outsource this

It is tempting to paste a problem description into an assistant and ask which technique fits, but the four-question map depends on facts that only you can supply: how stable your vocabulary actually is, how many labeled examples exist, and what near-zero marginal cost is worth to your business. The recommendation that comes back will sound confident either way, because assistants are trained toward answers that satisfy the person asking. Whatever your description omits, the model fills with assumptions you never stated and therefore cannot check.

The ticket-triage task from the opening scene will return throughout the book: Chapter 2 makes the first programmatic triage pass, Chapter 3 gives that triage a typed and validated output, Chapter 11 makes it the target of a routing workflow, and Chapter 25 deploys it as a service. Other chapters draw on other running examples, among them a corpus of annual reports for retrieval and a sentiment dataset for fine-tuning, but following one problem across the book’s capabilities is the clearest way to see how they fit together.

TipCost: what this chapter’s example actually costs

Era three is the first era in which inference itself carries a marginal cost. For the 10,000-ticket triage problem above, classifying every ticket with a lightweight model at typical 2026 rates costs well under one dollar, and because the batch is a full quarter’s volume, that figure is also the quarterly cost. Chapter 2 measures this cost precisely and introduces the two mechanisms that control it: prompt caching and model routing.

ImportantCompliance: where this chapter’s example is regulated

The same triage system, run over customer messages, processes personal information, and the rules that apply depend on where the customers are. Customers in the European Union place the data under the General Data Protection Regulation (GDPR), while the EU AI Act (the European Union’s risk-tiered rules for AI systems) applies separately and is triggered by the risk class of the system, irrespective of the data it processes. Customers in Colorado fall under a separate state framework. Appendix D provides the decision flowchart, and every later chapter that touches regulated data refers back to that appendix.

1.7 Exercises

The build labs and evaluate labs start in Chapter 2. Chapter 1 is conceptual, so its exercises are too.

1.7.1 Conceptual questions

  1. A team flags incoming insurance claims as “high risk” from free-text descriptions. The vocabulary in claims changes slowly, and they have 100,000 labeled historical claims. Which technique is the best starting point?

    1. Rules, since a slow-moving vocabulary can be encoded by hand once
    2. Traditional ML, since plentiful labels and a stable vocabulary suit it
    3. A generative system, since zero-shot capability makes the labels unnecessary
    4. A generative agent, since claims must be acted on beyond simply being flagged

  2. A generative system summarizes earnings transcripts. After three months in production, complaints arrive that the summaries miss material disclosures. Which failure best explains three months of undetected misses?

    1. The transcripts grew past the model’s context window during the quarter
    2. The provider deprecated the foundation model behind the system
    3. The team launched without a way to measure summary quality over time
    4. The prompt was tuned on press releases, which differ in form from transcripts

  3. A senior leader asks: “Now that we have generative models, do we still need data labeling?” The most accurate framing is:

    1. Yes. Labeled examples are still how we evaluate generative systems
    2. No. A generative model can label its own data without human review
    3. No. Labels were only ever needed when training supervised models
    4. Yes, but only for classification tasks with a fixed category list

  4. Which statement about the three eras is the least accurate?

    1. Generative systems can often handle tasks they were never trained for
    2. Traditional ML is usually cheaper per inference than a generative call
    3. Rule-based systems are usually easier to audit than generative ones
    4. Generative systems need no test set, since the model comes pre-evaluated

  5. You want to convince a sponsor that a generative ticket-triage system is worth its cost. Which framing is most persuasive?

    1. The model’s price per million tokens beats every major competitor
    2. The model places near the top of a widely cited public benchmark
    3. Triage costs X dollars per ticket against Y dollars of manual handling
    4. Response latency stays under two seconds at the 95th percentile under load

  6. Moving from rules to traditional ML to generative systems, which of these rises?

    1. The risk of a confident answer that is simply wrong
    2. The effort required to get a first working version running
    3. The brittleness of the system when vocabulary shifts
    4. The cost of labeling enough examples before starting

  7. In the generative era, where does the analyst’s effort concentrate?

    1. In writing and maintaining the regular expressions
    2. In hand-labeling thousands of training examples
    3. In tuning hyperparameters across training runs
    4. In framing the problem and evaluating the result

  8. The chapter places one capability in the “maybe new” category. Which one?

    1. Models can classify text in dozens of languages at once
    2. An analyst can build systems that act, going beyond systems that report
    3. Models can return JSON that conforms to a declared schema
    4. A company can fine-tune a foundation model on its own data

1.7.2 Build lab and evaluate lab

These begin in Chapter 2.

NoteWhere we go next

In Chapter 2 we make our first programmatic call to a model, route between a cheap model and a reasoning model based on the task, and use prompt caching to make the same prompt cost a fraction of itself the second time. The ticket-triage problem from the top of this chapter becomes a working five-line script by the end of the next one.

  1. Financial Times, “EY retracts study after researchers discover AI hallucinations,” May 2026. The underlying review, by the AI-detection firm GPTZero, found 16 of the report’s 27 cited sources fabricated, misattributed, or unreachable (https://gptzero.me/investigations/ey). Verified July 2026.↩︎