Appendix I — Appendix I: The Tooling Landscape

The tools below are worth knowing for each capability the book builds, grouped by where the work sits in the lifecycle from Chapter 1. This is the fastest-moving layer in the whole book, so we name every tool as a representative of its category, without endorsing it, and the list is a snapshot taken in 2026. For guidance on choosing a model or provider, which is a separate decision from choosing a tool, see Appendix E. Each part’s tool section points back here.

I.1 The model layer: gateways, structured output, and serving

Reaching a model. Nearly everything speaks the OpenAI API protocol, so the OpenAI Python SDK is the common client even for non-OpenAI models. OpenRouter (used in this book) puts many providers behind one key and one endpoint; LiteLLM and Portkey do the same as a proxy you run, adding routing, retries, and spend limits. Building against the shared protocol lets you change models later without rewriting calls, whereas building against a single provider’s quirks ties every call to that provider.

Getting structured output. Instructor (used in this book) wraps a model call so the reply validates against a Pydantic schema and retries when it does not; Outlines and lm-format-enforcer constrain generation so the tokens can only form valid output; most providers also offer a native JSON or structured-output mode. Whichever you use, validate the result, because a schema in the prompt is a request that the model may still violate.

Where the model runs. Hosted APIs (OpenAI, Anthropic, Google, reached through OpenRouter or directly) are the default, and hosted open-model inference (Together, Fireworks, Groq) runs open-weight models behind the same API so you get an open model without operating one. The cloud platforms (AWS Bedrock, Google Vertex AI, Azure AI) host models inside your existing cloud account and its data rules. To self-host an open model, a serving stack (vLLM, Ollama for a laptop, TGI, SGLang) presents it behind the same API; serverless GPU platforms (Modal, RunPod, Replicate, Baseten) run that stack per request so you do not keep a GPU warm. Chapter 4’s crossover decides between hosted and self-hosted on cost.

I.2 Documents: extraction, embeddings, and stores

Getting text out of documents. Open-weight options run from PyMuPDF, the fast default when a PDF’s text is already selectable, through pdfplumber for tables and Unstructured for ingesting many formats at once, to Docling and Marker, which use models to recover layout and reading order from messy or scanned pages. Hosted services include AWS Textract and Azure Document Intelligence, and the frontier vision-language models from Chapter 19 read a page image directly. Choose based on how much structure the answer depends on: plain text for prose, a layout-aware extractor when the numbers live in tables.

Embedding text. Open-weight models like BGE-M3 (used in this book), E5, GTE, and Nomic run locally, carry no per-call fee, and keep the text on your own machine. Hosted options like OpenAI’s text-embedding-3, Cohere Embed, and Voyage AI are a single API call with nothing to operate. Data residency favors the open-weight side, operational simplicity the hosted side. Whichever you choose, hold it fixed across a corpus, since changing it means re-embedding everything.

Storing and searching vectors. In-process stores like Chroma (used here), FAISS, and LanceDB live inside the application and need no server, handling up to low millions of vectors. Self-hosted servers like Qdrant, Milvus, and Weaviate add horizontal scale and richer filtering at the cost of running a service. Managed services like Pinecone and the hosted tiers of the others trade a bill for someone else owning scale and backups. pgvector keeps vectors inside Postgres, the pragmatic default when you already run one. Choose based on scale and how much operation you want to own, since benchmark leaderboards churn too quickly to guide the choice.

I.3 Retrieval and evaluation

Reranking. A cross-encoder rescores a shortlist after the first retrieval. Since the cross-encoder is too slow to read the whole corpus, rerank a shortlist of tens of candidates. BGE reranker v2-m3 (used in this book), Jina, and mixedbread rerankers run locally, while Cohere Rerank and Voyage offer the same as an API.

Keyword and hybrid search. Exact-term matching still catches what embeddings miss: rank_bm25 for a small corpus, Elasticsearch or OpenSearch at scale, and the built-in hybrid modes of Qdrant and Weaviate that blend dense and sparse scores. Learned-sparse models such as SPLADE sit between the two.

Evaluation and judges. Frameworks for scoring RAG and open-ended output include Ragas, DeepEval, TruLens, promptfoo, and the eval features of Phoenix, LangSmith, and Braintrust. They automate the mechanics of Chapter 9. The discipline still falls to you: pin the judge model and version, and validate the judge against human labels before you trust its scores.

I.4 Workflows, agents, and MCP

Orchestrating multi-step work. Plain Python handles most multi-step work, with the control flow you wrote. However, when the graph of steps grows, LangGraph, LlamaIndex Workflows, and Haystack model it explicitly, and durable engines such as Temporal or Prefect keep long-running flows alive across failures.

Agent runtimes. When the model decides the control flow, a runtime manages the tool-call loop. Common options are LangGraph, the OpenAI Agents SDK, Pydantic AI, smolagents, CrewAI, and AutoGen. All need a model trained for reliable tool use (Chapter 12) and a hard cap on steps so a confused agent cannot loop forever.

Model Context Protocol. MCP is the standard that lets an agent reach tools it did not ship with. FastMCP (used in this book) builds a server in a few lines; a growing catalog of reference servers exposes filesystems, databases, and SaaS APIs; and clients such as coding assistants and chat apps consume them. A tool that performs a destructive action still needs a human gate (Chapter 23), protocol or not.

I.5 Analytics and text-to-SQL

Clustering, projection, and topics. scikit-learn includes k-means, the classifiers, and PCA. UMAP and t-SNE handle projection, while HDBSCAN finds clusters without a fixed count. BERTopic combines the embedding, clustering, and labeling steps into a topic pipeline. None of these names the topics for you, which stays a human check (Chapter 13).

Text-to-SQL. The model writes the query, and a thin layer runs it and feeds back errors: the LangChain and LlamaIndex SQL helpers, or Vanna, which retrieves example queries to ground the model. In production, the guard is a semantic layer (dbt’s, Cube, or LookML) that exposes defined metrics by name, so that a metric nobody defined cannot be queried and the model never touches raw tables (Chapter 14).

I.6 Customizing models

Training a fine-tune. The Hugging Face stack (PEFT for LoRA, TRL for the training loop) is the base, with Axolotl, Unsloth, and Llama-Factory wrapping it in configuration and speed, while torchtune is PyTorch-native. Hosted fine-tuning (OpenAI, Together, Fireworks, Predibase) trades control for a managed job. Whether to fine-tune at all is the prior question (Chapter 15).

Quantization. To fit a larger model in less memory, bitsandbytes provides the 4-bit base for QLoRA, while GPTQ and AWQ produce quantized weights for serving, and llama.cpp’s GGUF format runs them on CPUs and laptops. Quantization trades a little accuracy for a much smaller footprint (Chapter 16).

Synthetic data. A capable model generates training examples, with Distilabel structuring the pipeline. However, keep real data in the mix and test on real data only, since training on model output across generations drifts from reality (Chapter 17).

I.7 Speech, vision, and time series

Speech. For transcription, Whisper and its faster reimplementations (faster-whisper, WhisperX), along with NVIDIA’s NeMo models such as Parakeet, run locally, while Deepgram, AssemblyAI, and OpenAI offer it hosted; pyannote handles speaker diarization. For the other direction, text-to-speech runs from open models (Kokoro, used here, Piper, Coqui) to hosted voices (ElevenLabs, OpenAI).

Vision. Vision-language models read images alongside text: open-weight Qwen-VL and InternVL, and the hosted frontier models (GPT, Claude, Gemini). They read printed text and labels closely and only estimate unlabeled quantities, so trust a count by eye accordingly (Chapter 19).

Time series. Forecasting foundation models predict a series they were never trained on: Chronos (used here), TimesFM, Moirai, and Lag-Llama run locally, and TimeGPT offers it as an API. Read the uncertainty band, since a single forecast line hides the risk (Chapter 20).

I.8 Knowledge graphs

Storing the graph. NetworkX (used in this book) holds a graph in memory, which is enough to learn and to prototype. For persistence and scale, Neo4j is the common property-graph database, with Memgraph, ArangoDB, and the embedded Kùzu as alternatives.

Building and querying with a model. Frameworks that extract a graph from text and traverse it for answers include Microsoft’s GraphRAG, LlamaIndex’s property-graph index, and the graph features of LangChain and Neo4j. Whatever extracts the edges, hold each one to the text it came from and reconcile entity names, or the graph fills with hallucinated relations and duplicate nodes (Chapter 21).

I.9 Production: serving, observability, and guardrails

Serving and interface. FastAPI (used in this book) wraps the system in an HTTP API; BentoML, LitServe, and Ray Serve add model-serving conveniences, the last for horizontal scale. For a chat interface a colleague can use, Chainlit (used here), Streamlit, and Gradio each stand one up quickly, and Open WebUI offers a ready chat front end.

Observability. To see what production is doing, a tracing tool like Langfuse, Arize Phoenix, LangSmith, Helicone, or the OpenTelemetry-based OpenLLMetry records every call’s inputs, outputs, cost, and latency. Without one, drift is invisible until a user reports it (Chapter 24).

Guardrails and gateways. Screening untrusted input and validating output is the job of NeMo Guardrails, Guardrails AI, Llama Guard, and injection-specific tools like Rebuff and Lakera. A gateway (LiteLLM, Portkey, Cloudflare AI Gateway) centralizes keys, spend limits, caching, and routing across the whole system.

--- title: "Appendix I: The Tooling Landscape" --- The tools below are worth knowing for each capability the book builds, grouped by where the work sits in the lifecycle from Chapter 1. This is the fastest-moving layer in the whole book, so we name every tool as a representative of its category, without endorsing it, and the list is a snapshot taken in 2026. For guidance on choosing a model or provider, which is a separate decision from choosing a tool, see [Appendix E](e-vendor-selection.qmd). Each part's tool section points back here. ## The model layer: gateways, structured output, and serving {#sec-tooling-model} **Reaching a model.** Nearly everything speaks the OpenAI API protocol, so the OpenAI Python SDK is the common client even for non-OpenAI models. OpenRouter (used in this book) puts many providers behind one key and one endpoint; LiteLLM and Portkey do the same as a proxy you run, adding routing, retries, and spend limits. Building against the shared protocol lets you change models later without rewriting calls, whereas building against a single provider's quirks ties every call to that provider. **Getting structured output.** Instructor (used in this book) wraps a model call so the reply validates against a Pydantic schema and retries when it does not; Outlines and lm-format-enforcer constrain generation so the tokens can only form valid output; most providers also offer a native JSON or structured-output mode. Whichever you use, validate the result, because a schema in the prompt is a request that the model may still violate. **Where the model runs.** Hosted APIs (OpenAI, Anthropic, Google, reached through OpenRouter or directly) are the default, and hosted open-model inference (Together, Fireworks, Groq) runs open-weight models behind the same API so you get an open model without operating one. The cloud platforms (AWS Bedrock, Google Vertex AI, Azure AI) host models inside your existing cloud account and its data rules. To self-host an open model, a serving stack (vLLM, Ollama for a laptop, TGI, SGLang) presents it behind the same API; serverless GPU platforms (Modal, RunPod, Replicate, Baseten) run that stack per request so you do not keep a GPU warm. Chapter 4's crossover decides between hosted and self-hosted on cost. ## Documents: extraction, embeddings, and stores {#sec-tooling-documents} **Getting text out of documents.** Open-weight options run from PyMuPDF, the fast default when a PDF's text is already selectable, through pdfplumber for tables and Unstructured for ingesting many formats at once, to Docling and Marker, which use models to recover layout and reading order from messy or scanned pages. Hosted services include AWS Textract and Azure Document Intelligence, and the frontier vision-language models from Chapter 19 read a page image directly. Choose based on how much structure the answer depends on: plain text for prose, a layout-aware extractor when the numbers live in tables. **Embedding text.** Open-weight models like BGE-M3 (used in this book), E5, GTE, and Nomic run locally, carry no per-call fee, and keep the text on your own machine. Hosted options like OpenAI's text-embedding-3, Cohere Embed, and Voyage AI are a single API call with nothing to operate. Data residency favors the open-weight side, operational simplicity the hosted side. Whichever you choose, hold it fixed across a corpus, since changing it means re-embedding everything. **Storing and searching vectors.** In-process stores like Chroma (used here), FAISS, and LanceDB live inside the application and need no server, handling up to low millions of vectors. Self-hosted servers like Qdrant, Milvus, and Weaviate add horizontal scale and richer filtering at the cost of running a service. Managed services like Pinecone and the hosted tiers of the others trade a bill for someone else owning scale and backups. pgvector keeps vectors inside Postgres, the pragmatic default when you already run one. Choose based on scale and how much operation you want to own, since benchmark leaderboards churn too quickly to guide the choice. ## Retrieval and evaluation {#sec-tooling-retrieval} **Reranking.** A cross-encoder rescores a shortlist after the first retrieval. Since the cross-encoder is too slow to read the whole corpus, rerank a shortlist of tens of candidates. BGE reranker v2-m3 (used in this book), Jina, and mixedbread rerankers run locally, while Cohere Rerank and Voyage offer the same as an API. **Keyword and hybrid search.** Exact-term matching still catches what embeddings miss: rank_bm25 for a small corpus, Elasticsearch or OpenSearch at scale, and the built-in hybrid modes of Qdrant and Weaviate that blend dense and sparse scores. Learned-sparse models such as SPLADE sit between the two. **Evaluation and judges.** Frameworks for scoring RAG and open-ended output include Ragas, DeepEval, TruLens, promptfoo, and the eval features of Phoenix, LangSmith, and Braintrust. They automate the mechanics of Chapter 9. The discipline still falls to you: pin the judge model and version, and validate the judge against human labels before you trust its scores. ## Workflows, agents, and MCP {#sec-tooling-agents} **Orchestrating multi-step work.** Plain Python handles most multi-step work, with the control flow you wrote. However, when the graph of steps grows, LangGraph, LlamaIndex Workflows, and Haystack model it explicitly, and durable engines such as Temporal or Prefect keep long-running flows alive across failures. **Agent runtimes.** When the model decides the control flow, a runtime manages the tool-call loop. Common options are LangGraph, the OpenAI Agents SDK, Pydantic AI, smolagents, CrewAI, and AutoGen. All need a model trained for reliable tool use (Chapter 12) and a hard cap on steps so a confused agent cannot loop forever. **Model Context Protocol.** MCP is the standard that lets an agent reach tools it did not ship with. FastMCP (used in this book) builds a server in a few lines; a growing catalog of reference servers exposes filesystems, databases, and SaaS APIs; and clients such as coding assistants and chat apps consume them. A tool that performs a destructive action still needs a human gate (Chapter 23), protocol or not. ## Analytics and text-to-SQL {#sec-tooling-analytics} **Clustering, projection, and topics.** scikit-learn includes k-means, the classifiers, and PCA. UMAP and t-SNE handle projection, while HDBSCAN finds clusters without a fixed count. BERTopic combines the embedding, clustering, and labeling steps into a topic pipeline. None of these names the topics for you, which stays a human check (Chapter 13). **Text-to-SQL.** The model writes the query, and a thin layer runs it and feeds back errors: the LangChain and LlamaIndex SQL helpers, or Vanna, which retrieves example queries to ground the model. In production, the guard is a semantic layer (dbt's, Cube, or LookML) that exposes defined metrics by name, so that a metric nobody defined cannot be queried and the model never touches raw tables (Chapter 14). ## Customizing models {#sec-tooling-customizing} **Training a fine-tune.** The Hugging Face stack (PEFT for LoRA, TRL for the training loop) is the base, with Axolotl, Unsloth, and Llama-Factory wrapping it in configuration and speed, while torchtune is PyTorch-native. Hosted fine-tuning (OpenAI, Together, Fireworks, Predibase) trades control for a managed job. Whether to fine-tune at all is the prior question (Chapter 15). **Quantization.** To fit a larger model in less memory, bitsandbytes provides the 4-bit base for QLoRA, while GPTQ and AWQ produce quantized weights for serving, and llama.cpp's GGUF format runs them on CPUs and laptops. Quantization trades a little accuracy for a much smaller footprint (Chapter 16). **Synthetic data.** A capable model generates training examples, with Distilabel structuring the pipeline. However, keep real data in the mix and test on real data only, since training on model output across generations drifts from reality (Chapter 17). ## Speech, vision, and time series {#sec-tooling-modalities} **Speech.** For transcription, Whisper and its faster reimplementations (faster-whisper, WhisperX), along with NVIDIA's NeMo models such as Parakeet, run locally, while Deepgram, AssemblyAI, and OpenAI offer it hosted; pyannote handles speaker diarization. For the other direction, text-to-speech runs from open models (Kokoro, used here, Piper, Coqui) to hosted voices (ElevenLabs, OpenAI). **Vision.** Vision-language models read images alongside text: open-weight Qwen-VL and InternVL, and the hosted frontier models (GPT, Claude, Gemini). They read printed text and labels closely and only estimate unlabeled quantities, so trust a count by eye accordingly (Chapter 19). **Time series.** Forecasting foundation models predict a series they were never trained on: Chronos (used here), TimesFM, Moirai, and Lag-Llama run locally, and TimeGPT offers it as an API. Read the uncertainty band, since a single forecast line hides the risk (Chapter 20). ## Knowledge graphs {#sec-tooling-graphs} **Storing the graph.** NetworkX (used in this book) holds a graph in memory, which is enough to learn and to prototype. For persistence and scale, Neo4j is the common property-graph database, with Memgraph, ArangoDB, and the embedded Kùzu as alternatives. **Building and querying with a model.** Frameworks that extract a graph from text and traverse it for answers include Microsoft's GraphRAG, LlamaIndex's property-graph index, and the graph features of LangChain and Neo4j. Whatever extracts the edges, hold each one to the text it came from and reconcile entity names, or the graph fills with hallucinated relations and duplicate nodes (Chapter 21). ## Production: serving, observability, and guardrails {#sec-tooling-production} **Serving and interface.** FastAPI (used in this book) wraps the system in an HTTP API; BentoML, LitServe, and Ray Serve add model-serving conveniences, the last for horizontal scale. For a chat interface a colleague can use, Chainlit (used here), Streamlit, and Gradio each stand one up quickly, and Open WebUI offers a ready chat front end. **Observability.** To see what production is doing, a tracing tool like Langfuse, Arize Phoenix, LangSmith, Helicone, or the OpenTelemetry-based OpenLLMetry records every call's inputs, outputs, cost, and latency. Without one, drift is invisible until a user reports it (Chapter 24). **Guardrails and gateways.** Screening untrusted input and validating output is the job of NeMo Guardrails, Guardrails AI, Llama Guard, and injection-specific tools like Rebuff and Lakera. A gateway (LiteLLM, Portkey, Cloudflare AI Gateway) centralizes keys, spend limits, caching, and routing across the whole system.