Claude Agent SDK in Production, Part 13: Trust, but Verify: Structured Outputs, Budgets, and Evals

Twelve parts in, your analyst is capable, governed, durable, and contained. It reaches databases and the open web, pauses on approval cards, logs every move, and survives a refresh. And every number it hands you is still one you check by eye. "It looked right in the demo" is where most agent projects quietly stop, and it is not a quality bar you can ship. This part is the answer to the question every production deployment eventually gets asked: how do you know it's good? Three mechanisms, escalating in strength. Make the output machine-checkable. Make the cost bounded. Make the quality measurable. By the end, when you change a prompt, a number tells you whether the analyst got better or worse, and you have the receipts.

That figure is the destination. Every column is a real run of the suite we build in this part, and every red cross is a number the analyst got wrong when I removed a single sentence from its prompt. Let's earn it, starting with the smallest of the three mechanisms and the one everything else rests on.

The wall of prose problem

Ask the analyst a question and it answers beautifully, for a human. Headline, a table, a paragraph of interpretation. Now imagine another service wants that answer: a dashboard that plots the key metric, a Slack bot that posts the headline, a pipeline that files the caveats. All of them have to parse English to get there, and English is exactly the thing that breaks the moment the model phrases things differently. The answer is a wall of prose, and your code can't read walls.

The fix is to ask for a form instead of an essay. Not instead of the prose, alongside it: the agent still works free-form, still writes its markdown reply, but the run now ends with a structured object that matches a schema you defined. Here is that schema, a plain Pydantic model:

class Metric(BaseModel):
    label: str = Field(description="What this number is, e.g. 'March revenue, Downtown'.")
    value: float = Field(
        description="The number itself, from a query result or script output. "
        "Plain digits: no currency symbols, no thousands separators."
    )
    unit: str = Field(description="The unit, e.g. 'USD', '%', 'units'.")


class AnalysisSummary(BaseModel):
    """What every analysis boils down to, as a form instead of an essay."""

    headline: str = Field(description="The finding in one sentence, numbers included.")
    key_metrics: list[Metric] = Field(
        description="Every number the answer depends on, one metric each."
    )
    caveats: list[str] = Field(
        description="Data-quality flags, assumptions made, questions not answered. "
        "Empty list if there are none; never invent one to fill space."
    )
    chart_paths: list[str] = Field(
        description="Workspace-relative paths of chart files created this turn, if any."
    )

Look at those description strings. They are not documentation, they are prompt. Part 6 taught that a tool description is read by the model; a schema field description is the same thing, read as the agent fills the form. "Plain digits: no currency symbols" is an instruction, and the agent follows it.

Wiring it into the run is one option on ClaudeAgentOptions. You hand the SDK the JSON schema, generated straight from the model so the contract lives in exactly one place:

        # Part 13: the contract. The agent still works free-form (tools,
        # prose, charts), but the RUN now ends with a summary matching
        # this schema, delivered on ResultMessage.structured_output. The
        # schema is generated from the Pydantic model, so the contract
        # lives in exactly one place.
        output_format={
            "type": "json_schema",
            "schema": AnalysisSummary.model_json_schema(),
        },

Under the hood, the engine collects the summary through a tool call it makes at the end of the turn, named StructuredOutput, whose input is the object. It arrives two ways at once: on ResultMessage.structured_output as a dict, and on ResultMessage.result as the same JSON stringified. We only need the dict.

The vocabulary's final extension

That summary has to reach the browser, and here the series collects a debt it has carried since Part 2. Our whole frontend speaks one event vocabulary, defined once and only ever extended. The complete event, which has carried usage and total_cost_usd since Part 2, gains one field:

            elif isinstance(message, ResultMessage):
                event = {
                    "type": "complete",
                    "usage": message.usage,
                    "total_cost_usd": message.total_cost_usd,
                    "duration_ms": message.duration_ms,
                }
                # The vocabulary's last extension. model_validate is the
                # boundary check: the engine already enforced the schema,
                # this makes a broken contract fail HERE, loudly, instead
                # of three services downstream.
                if message.structured_output is not None:
                    summary = AnalysisSummary.model_validate(message.structured_output)
                    event["structured_output"] = summary.model_dump()
                if message.subtype in STOP_REASONS:
                    event["stop_reason"] = STOP_REASONS[message.subtype]
                yield event

model_validate is doing quiet, important work. The engine already enforced the schema at generation time, so this rarely fails, but it's the boundary check: if the contract is ever broken, it fails here, loudly, at the edge of your server, instead of silently three services downstream. Validate at the boundary and everything past it can trust the shape.

And that StructuredOutput tool call the engine makes? It would otherwise render as a meaningless badge in the transcript. So the translator hides it, exactly the way Part 10 hid the card-backed tools: its story is told by the receipt, not a badge.

# Tool calls whose story is told by a gate-emitted card instead of a badge.
CARD_BACKED_TOOLS = {"AskUserQuestion", "ExitPlanMode"}

# Part 13: the engine collects the structured summary through a tool call
# named StructuredOutput whose input IS the summary object. Its story is
# told by the complete event, so like the card-backed tools it gets no
# badge; unlike them, nothing pauses, so there's no card either.
RECEIPT_BACKED_TOOLS = {"StructuredOutput"}

# ResultMessage subtypes worth naming on the wire: the run was cut short
# by a cap you set, not finished and not crashed. Everything else keeps
# its existing story (success, or the interrupt flow from Part 9).
STOP_REASONS = {
    "error_max_budget_usd": "max_budget",
    "error_max_turns": "max_turns",
}

Stop and count what just happened to the event vocabulary, because this is the last time we touch it. Part 2 defined six event types. Since then we have added six more types, one envelope field (the SSE id:), and now three optional payload fields, and the parser your frontend runs, the switch (event.type) that has ignored what it doesn't recognize since Part 3, has never once changed. That was the bet in Part 2: design the envelope, not the message, and you can extend it for a year without a rewrite. Here is the bet paid in full:

On the frontend, the summary rides onto the assistant turn and renders as a small card under the prose: the headline, the metrics as a label-and-value ledger, the caveats beneath a dashed line. The TypeScript type mirrors the Pydantic model field for field, because the wire is the contract between them:

export type Metric = { label: string; value: number; unit: string };

export type AnalysisSummary = {
  headline: string;
  key_metrics: Metric[];
  caveats: string[];
  chart_paths: string[];
};

Here it is in a real run, the Q1-vs-Q2 comparison, with the summary card sitting right under the prose answer, both saying the same numbers in two languages, one for you and one for your code:

The prepaid meter

The cost ritual has run since Part 1: print total_cost_usd after every turn so cost is a number you watch, not a surprise you get. Watching is good. It is also passive. A runaway analysis, a question that sends the agent down forty tool calls of assumptions, spends the money first and shows you the receipt after. Watching graduates to enforcing here, with one more option:

        # Part 13: the prepaid meter. None means unmetered (the default);
        # a number is a hard stop enforced by the engine, not a request.
        # The run that crosses the line ends with subtype
        # error_max_budget_usd and no final answer, so treat this as a
        # circuit breaker for runaways, not a precision instrument.
        max_budget_usd=budget_usd,

Think of it as a prepaid meter. You load two cents; when the run has spent two cents, the power cuts. It rides in from the request, one new field on the ChatRequest the app has grown part by part:

class ChatRequest(BaseModel):
    message: str
    workspace_id: str | None = None
    session_id: str | None = None  # the memory switch: absent = fresh start
    mode: Literal["ask", "plan"] = "ask"  # plan = propose before touching anything
    thinking: bool = False  # extended thinking: a visible, billed scratchpad
    # Part 13: the prepaid meter, per request. None = unmetered. gt=0
    # because a zero budget is a run that can't start, which is Stop.
    budget_usd: float | None = Field(default=None, gt=0, le=5.0)

Now give a hard question a genuinely stingy budget and watch it fail gracefully. I asked for a full performance report, every store, every product, every month, on a two-cent cap. Here is exactly what happened on the wire:

Two things in that trace are worth their own sentence. First, the overshoot: the cap was two cents, the run cost $0.0258. max_budget_usd is a circuit breaker, not a laser. It checks between API rounds, so you pay for the round that crossed the line, and the first round of an agent turn (the fat system prompt, the cache creation) is the most expensive one. On a leaner spike run I measured the overshoot at a single cent ($0.0207 against a two-cent cap). Budget for the breaker tripping a little late. Second, and lovely: the model narrated the approaching limit. "I'm approaching my budget limit." The engine tells the agent about the meter, so it spends its last cents trying to wrap up rather than getting cut off mid-thought.

A stopped run still needs a story on the wire, and the complete event carries it in that stop_reason field you saw added earlier. In the UI, the receipt reads budget exhausted instead of a normal cost line, and any tool badge still spinning when the plug was pulled gets settled so the turn reads as finished, not frozen:

The driving test

Here's the scenario that makes all of this matter. Back in Part 6 we added a house rule: "never do arithmetic in your head." In Part 10 we rewrote another rule three times. Every one of those was a change to the prompt, and every time the honest question was: did that make the analyst better, or did it just make the one example I tried look better? Vibes cannot answer that. You need a driving test: not a feeling, a pass rate.

An eval suite is three pieces. A set of cases with known-correct answers. A runner that puts the real agent through each one. And a judge that grades what comes back. Let's build all three, tutorial-scale but real, and the whole thing works only because the Beanline data is deterministic: the generator builds the same database every time, so "the right answer" is a fact I verified with SQL, not a snapshot that drifts.

The cases are a YAML file. Each is a question plus the facts a correct answer must contain, and two of them are built around the analyst's known weak spots:

- id: portland-share
  question: >
    What percentage of total company revenue in the first half of 2026
    came from the Portland stores? Name the stores you counted as
    Portland.
  expected:
    - >
      The Portland stores are Downtown (S01), Airport (S02), and
      Old Town (S06).
    - >
      The share is 57.8% (Portland 713,237.80 of 1,234,310.20 total;
      anything from 57 to 58 percent passes, anything outside fails).

That portland-share case is not academic. Across building this series I logged the analyst confidently botching exactly this kind of share-of-total question four separate times: 63% against a true 57.8%, 63% against a true 59.0%, 38% against a true 48.9%. A percentage is arithmetic the model is tempted to eyeball, and eyeballing is where it slips. The other planted case, per-store-plus-total, is the six-number sum that tempts it to add in its head, plus the duplicate row baked into the sample data since the beginning. The cases are designed around the failures, not the successes.

The runner puts the real agent through each case. Same build_options the app ships, same prompt, same tools, same sandbox. The only substitution is a robot at the approval gate, because the suite runs unattended on a throwaway desk it's allowed to burn down:

async def robot_approver(tool_name, tool_input, context):
    """The suite's stand-in for your Approve click. It can say yes to
    everything because eval cases are known questions on deterministic
    data, in throwaway workspaces, with the sandbox still on."""
    return PermissionResultAllow()

Each attempt is one fresh workspace, one real run, one graded verdict, and note the two safety rails: the per-attempt max_budget_usd (the same option from earlier, doing double duty so a misbehaving prompt can't turn a suite run into a bill) and a semaphore so attempts run concurrently without launching thirty agents at once:

async def run_attempt(case: dict, semaphore: asyncio.Semaphore) -> Attempt:
    """One case, one fresh desk, one real agent run, one graded verdict."""
    async with semaphore:
        workspace = workspace_path(create_workspace())
        client = ClaudeSDKClient(
            options=build_options(
                workspace, None, robot_approver, budget_usd=ATTEMPT_BUDGET_USD
            )
        )
        prose: list[str] = []
        summary, cost, stop = None, 0.0, "unknown"
        started = time.monotonic()
        try:
            await client.connect()
            await client.query(case["question"])
            async for message in client.receive_response():
                # Main-agent text only: a subagent's words carry
                # parent_tool_use_id, and the judge grades the answer
                # the user would have seen.
                if isinstance(message, AssistantMessage) and not message.parent_tool_use_id:
                    prose += [b.text for b in message.content if isinstance(b, TextBlock)]
                elif isinstance(message, ResultMessage):
                    summary = message.structured_output
                    cost = message.total_cost_usd or 0.0
                    stop = message.subtype

The judge eats its own dog food

How do you grade a free-form answer against a list of expected facts? A number and its rounding, a percentage and its tolerance, a fact that might be in the prose or the summary card? Exact string matching is hopeless. This is the job LLM-as-judge exists for: a second model call that reads the answer and the ground truth and returns a verdict. And the verdict is itself a structured output, using the exact mechanism we built at the top of this part. The evals eat their own dog food; a judge that returned prose would need a judge of its own.

JUDGE_PROMPT = """You grade an AI data analyst's answer against known-correct facts.

Pass the answer only if EVERY expected fact is present and numerically
correct, in the prose or in the structured summary. Formatting never
matters: $51,319.60 and 51319.6 and "about $51.3K" all match 51,319.60,
and rounding to one decimal place is fine. A wrong number, a missing
fact, or a contradiction fails. Where an expected fact states its own
tolerance, apply exactly that tolerance.

Judge facts, not style. Extra detail, ordering, tone, length, and
chart choices are all irrelevant. You are checking arithmetic against
ground truth, not reviewing prose."""


class Verdict(BaseModel):
    """The grade, as a form. `failures` is the part you'll actually read."""

    passed: bool = Field(description="True only if every expected fact checks out.")
    failures: list[str] = Field(
        description="One entry per expected fact that is missing or wrong, "
        "quoting what the answer said instead. Empty when passed."
    )

The judge call gives the model no tools at all: it compares text, it doesn't need hands. It gets the question, the expected facts, the analyst's prose, and the analyst's summary, and returns the Verdict, validated the same way every structured output in this part is validated:

    options = ClaudeAgentOptions(
        tools=[],  # the judge compares text; it gets no hands on purpose
        model=JUDGE_MODEL,
        system_prompt=JUDGE_PROMPT,
        output_format={"type": "json_schema", "schema": Verdict.model_json_schema()},
    )
    async for message in query(prompt=prompt, options=options):
        if isinstance(message, ResultMessage):
            verdict = Verdict.model_validate(message.structured_output)
            return verdict, message.total_cost_usd or 0.0
    raise RuntimeError("The judge never returned a verdict.")

Run it, and you get a pass-rate table. My first honest suite run scored 6 of 7, and the one failure was real: on the duplicate-row case, the analyst reported the deduplicated March total as $206,815.80 when the right answer is $206,728.30, and the judge caught it and quoted the mistake. That is the suite working: an eval you build finds a bug you'd otherwise ship.

The regression, on purpose

Now the payoff, and the reason this whole part exists. I have a suite that gives me a number. Let me use it to answer the question I couldn't answer before: does the "never do arithmetic in your head" rule actually matter, or is it superstition?

Science needs a control, so here's the protocol: run the 8-case suite three times over, at three attempts per case (24 attempts total). First with the rule in place. Then I delete the rule, one sentence, and run the identical suite. Then I restore it and run once more. Same data, same judge, same everything except one line of the prompt. These are all real runs; the numbers are from evals/runs/.

Run 1, rule in place: 24 out of 24. A clean 100%. Every case, every attempt, green. Agent spend $1.4366, 179 seconds wall.

Then I removed exactly this line from the system prompt:

Every number you state must come from a query result or script output.
Never do arithmetic in your head, not even totals.

Run 2, rule removed: 20 out of 24. 83%. Four attempts failed, and every single failure is arithmetic the model did in its head instead of in SQL. The portland-share case came back 38.1% against the true 57.8%, the same class of error I'd logged three times before, now reproduced on command. The fastest-growth case reported 43.5% growth instead of 70.6%, because the model used quarterly numbers where the question said monthly. And per-store-plus-total failed twice out of three attempts, hand-summing the six stores to $1,234,710.20 and $1,234,710.30 when the real total is $1,234,310.20. Off by four hundred dollars, twice, in different directions, exactly the way head-arithmetic goes wrong.

Then I put the one line back.

Run 3, rule restored: 24 out of 24. Back to 100%. Agent spend $1.3759.

That is the figure at the top of this part, and it is the entire argument for evals in three columns. Without the suite, "never do arithmetic in your head" is a belief. With it, it's a measured 17-point swing in accuracy, with the exact wrong numbers on record. When you change a prompt now, you don't argue about whether it helped. You run the suite and read the number. Part 10 called prompt engineering probabilistic, not mechanical: write the rule, measure the rate, keep the receipts. This is the measuring, and it is the safety net under every prompt change you'll ever make.

What production adds

What we built is the honest core of an eval harness, and it is genuinely useful, but a production system layers more furniture on the same skeleton, and it's worth naming so you can read that code when you meet it. Cases and suites become database rows, not a YAML file, so you can track them over time. Each case runs as multiple attempts with the pass rate stored per attempt, because a stochastic model needs a distribution, not a single coin flip. The runner grows per-attempt timeouts and heartbeats so a hung agent doesn't wedge the whole suite. The judge's full transcript gets stored, not just its verdict, so a disputed grade can be audited. And the results feed a trend line across runs, so "did this week's prompt changes help?" is a chart, not a memory. The reference app this series is modeled on runs exactly this shape at scale. You now understand its core well enough to read the rest.

The cost ritual

All real runs from building this part, on claude-haiku-4-5. The eval suite is the new heavyweight line, and it's worth seeing what a driving test costs to sit:

Two things stand out. The budget overshoot is real and small: $0.0258 against a two-cent cap. And a full suite run costs about a dollar-fifty in agent spend plus a dime for the judge, which is the actual price of the confidence in that top figure. That is cheap for the ability to answer "did this prompt change help?" with a number instead of a shrug. The judge is a tenth of the bill precisely because it's a cheap model doing a narrow, tool-free job, which is the whole design.

Commit it, from the project root:

git add backend frontend
git commit -m "part 13: structured outputs, budgets, and an eval suite"

Your analyst is now provably good, on your terms: it hands back machine-checkable summaries, it can't blow a budget, and when you change a prompt a pass rate tells you whether you helped or hurt. It is, by every measure this series set out to build, production-grade software. There is exactly one thing left, and it's the most physical lesson in either series. Everything you've built runs on your laptop. The SDK spawns a subprocess, keeps state on disk, and holds streams open for minutes, and every one of those breaks the moment you try to put it on the serverless platforms most tutorials end on. Next, the last part: we put the whole thing on a real server, a Hetzner VM with HTTPS that survives a reboot, and we do it by actually deploying it.

The complete, tested code for this part lives in part-13-structured-evals in the companion repo. Code blocks with a GitHub icon link straight to the exact file; "View full file" shows the whole file in place with this section's changes highlighted.