Codex App Server in Production, Part 10: Plans, Questions, and the Reasoning Dial

Mid-build, unprompted by any UI of ours, the agent stopped. Its tool call froze on the wire, a JSON-RPC request with an id slid up the pipe, and inside it: three cafe color palettes, each with a label and a one-line description, waiting for a human. We picked "Midnight Velvet". The turn thawed, and the site that landed was charcoal, caramel, and muted rose, exactly as described. That round trip is real, captured, and about a page of code, because the machinery it needs has existed since Part 2.

Act III opens here. Acts I and II built a builder and put you in control of it; this act gives it judgment you can see. By the end of this page, Pagewright proposes a written plan before touching the disk, ticks off its own checklist while it builds, asks structured questions instead of guessing, and thinks exactly as hard as each message deserves, with Part 8's meter showing what the thinking cost.

Look at the receipt under the plan: blueprint (read-only). That label is not decoration. The planning turn ran inside a sandbox that refuses writes at the OS level, which means "propose first" stopped being a request and became a guarantee. Getting there honestly, though, requires admitting that the protocol feature this part was originally designed around no longer exists. That story first.

The guess you paid for

Every builder you have met so far in this series swings on the first message. Hand it the Beanline brief and it reads, reasons, and produces index.html in one motion. For a one-page cafe site, that eagerness is charming. Now imagine the brief is ambiguous, the client is picky, and the build burns 150k tokens before anyone checks the direction. An eager guess at that scale is not charming; it's an invoice.

The sibling series named this problem the anatomy of an expensive guess and spent a part fixing it with plan mode and structured questions. If you want the from-zero treatment of the UX pattern, read that page; it earns the pattern one design question at a time, and nothing here re-teaches it. This part builds the Codex-shaped version, and the protocol shapes underneath turn out to be almost entirely new material: a mode that vanished, a tool that must be switched on twice, and a server request that politely warns you it is under development.

Blueprint mode: a turn that cannot touch the disk

Here's the honesty the concepts page has been promising since the series began. Older protocol versions had a collaborationMode param on turn/start; the 0.139-era reference app this series is drawn from still sends one. On our pinned 0.142.4, it is gone. The mode concept survives only as a setting in Codex's own TUI; there is no "plan mode" to ask the wire for.

So what is a blueprint, if the protocol won't name one? A blueprint is a turn that cannot write. You already own everything needed to say that precisely, because Part 6 taught the wristbands: send the turn out with the look-only policy and the OS itself enforces "propose, don't build". No mode enum, no trust in the prompt, no way for an enthusiastic model to helpfully start early. This is a stronger promise than collaborationMode ever made, and it costs three fields.

The API grows one flag and two dials. None means "the default", so every client from Parts 3 through 9 keeps working unchanged:

class ChatRequest(BaseModel):
    message: str
    # Part 10: the consultative dials. plan_first sends the turn out
    # read-only with the blueprint nudge; effort and summary ride
    # turn/start as-is. None means "the default", so old clients keep
    # working unchanged.
    plan_first: bool = False
    effort: str | None = None
    summary: str | None = None

And the chat endpoint's turn/start learns to dress a blueprint turn differently (ignore effort and summary for now; they get their own section):

    started = await client.request("turn/start", {
        "threadId": thread_id,
        # A blueprint turn carries the nudge INSIDE the message: the
        # sandbox guarantees "no files", the nudge asks for the plan
        # instead of a narration about not being able to write.
        "input": [{"type": "text", "text":
                   (req.message + BLUEPRINT_NUDGE) if req.plan_first
                   else req.message}],
        "sandboxPolicy": ({"type": "readOnly"} if req.plan_first
                          else sandbox_policy(mode, workspace)),
        "approvalPolicy": "never" if req.plan_first else approval_policy(mode),
        # The dials, per turn. summary "detailed" has been the setting
        # since Part 3 (without it the reasoning drawer stays empty);
        # now both are the sender's choice, message by message.
        "effort": effort,
        "summary": summary,
    })

Two decisions in that fence deserve slow reading. The sandboxPolicy override is the whole feature: a plan_first turn ships {"type": "readOnly"} regardless of the project's mode, and approvalPolicy drops to "never" because there is nothing to escalate when nothing can be written. And the nudge rides inside the message, not in the policy:

# The blueprint nudge, appended to the user's message on plan_first
# turns. The read-only sandbox already guarantees "no files"; the nudge
# is so the agent PLANS instead of narrating that it cannot write.
BLUEPRINT_NUDGE = ("\n\nPropose a numbered build plan first; "
                   "do not write any files in this turn.")
PLAN_TOOL_NUDGE = (
    "When a task involves more than one step, maintain a live plan with "
    "the update_plan tool: set it before you start building and update "
    "step statuses as you complete them.")

Why both a wall and a request? Because they do different jobs. The sandbox guarantees no files change; it cannot make the model plan. Without the nudge, a read-only turn tends to produce a shrug: an explanation that it would love to build but cannot write, would you like it to describe what it might do. The nudge turns containment into intent. The wall keeps it honest; the words give it a job. (The second constant, PLAN_TOOL_NUDGE, belongs to the next section; it's here because they live together at the top of the file.)

Here's the whole flow from the real capture, both turns:

Run it against the Beanline brief with the toggle armed, and the wire does exactly what the diagram claims. The agent reads brief.md, the copy deck, and the logo, narrating as it goes. Then the plan arrives, and here is the second honesty beat of the part: it arrives as prose. Numbered markdown, streaming through text_delta like any answer since Part 3. Eight steps for Beanline: inline the logo SVG to honor the single-file rule, set CSS variables for the brand colors, structure four semantic sections, one section per plan item, ending with "If you want, I can next turn this plan into the actual index.html." The capture's tallies: 701 text deltas, zero plan notifications, zero file changes, and the file tree did not move. The blueprint is a document you read, not a data structure, and pretending otherwise would be this part inventing a protocol that isn't there.

Approval, then, needs no new endpoint, no decision enum, no card. You read the plan; if you like it, you say so. The frontend arms exactly one turn per toggle (the pill disarms itself after sending, so the follow-up naturally goes out at the project's own mode), and "Build it." is an ordinary message on the same thread. The model remembers the plan it wrote one turn ago; two words are enough to cash it in. Fifty-four seconds later there's a 594-line index.html honoring all eight numbered steps.

Checkpoint. Right now you have: a toggle that sends one turn out read-only with a planning nudge, a receipt that names the posture, and approval that costs two words. What you don't have yet is any machine-readable trace of plans. That changes on the build turn.

The living checklist

Watch the build turn from the dessert shot again: above the composer, a panel titled "The builder's plan", four steps, two struck through, one pulsing. Nothing in our backend invented those steps. The engine has a plan tool (update_plan): the agent calls it to declare its own step list and to update statuses as it works, and every call surfaces as a turn/plan/updated notification. That's the written estimate taped to the workshop window, ticking itself off.

Getting it to fire took two switches, and discovering that took most of a morning, so here is the truth in one place. Switch one is thread-level config: include_plan_tool hands the agent the tool. Verified live: the flag alone changes nothing. Four traces, zero plan notifications; the model never volunteers the tool, it builds. Switch two is a standing instruction in developerInstructions (the PLAN_TOOL_NUDGE you saw above), and with both set, build turns emit the checklist reliably. Both ride thread/start:

    started = await client.request("thread/start", {
        "cwd": str(workspace),
        "sandbox": "workspace-write",
        "approvalPolicy": "never",
        "model": MODEL,
        "config": {
            "include_plan_tool": True,
            "features.default_mode_request_user_input": True,
        },
        "developerInstructions": PLAN_TOOL_NUDGE,
    })

(The second config key is the next section arriving early; thread switches travel in pairs here because thread/start happens once per conversation.)

The notification's shape is friendly: the whole plan, re-sent in full on every change, {plan: [{step, status}], explanation} with statuses pending, inProgress, completed. Note the camelCase inProgress; the protocol's enum spelling, worth copying exactly into your types. The translation is one more arm on the Part 2 envelope, eight parts old and still absorbing new event types without a redesign:

    if method == "turn/plan/updated":
        return {"type": "plan_update",
                "steps": [{"step": s.get("step", ""),
                           "status": s.get("status", "")}
                          for s in p.get("plan", [])],
                "explanation": p.get("explanation")}
    if method == "item/plan/delta":
        # In the schema, never in a live 0.142.4 trace (the checklist
        # arrives whole via turn/plan/updated). Translated so a CLI that
        # starts streaming plan text doesn't get dropped on the floor.
        return {"type": "plan_delta", "item_id": p.get("itemId", ""),
                "text": p.get("delta", "")}

That second branch is schema archaeology, disclosed rather than hidden: item/plan/delta exists in the generated schema, presumably for streaming plan text, and it never fired in any trace we captured. We translate it anyway, so a future CLI that starts using it degrades gracefully instead of dropping frames.

Because plan_update lands in Part 9's event log like everything else, the frontend's job is four lines, and the whole-plan-every-time shape is why. Last write wins; replay into a fresh tab reconstructs the current checklist for free:

      } else if (event.type === "plan_update") {
        setPlan({ steps: event.steps, explanation: event.explanation });

Now the part of this section you should trust because it's untidy. The plan tool is stochastic. On build turns with both switches set, it fired in every capture, but the step granularity varies run to run: sometimes three steps, sometimes five for the same brief. On read-only blueprint turns it usually stays silent, which fits its nature (the tool tracks progress, and a blueprint turn produces no progress to track), yet one live run cheerfully used it to track the planning legwork itself: read the brief, inspect assets, draft the plan. If your product design requires a checklist on every turn, this tool will disappoint you. Pagewright's design absorbs the variance instead: the panel renders when a plan exists and renders nothing when none does, and its absence is a normal state, not a bug. The estimate taped to the window is the builder's own habit, not a form you made it fill in.

Checkpoint. Right now you have: blueprints as readable prose on read-only turns, and on build turns, the agent's own step list ticking pending → inProgress → completed in every tab, replay included. Time for the feature this part opened with: the turn that stops and asks.

The question that freezes a tool call

Everyone has lived both versions of the contractor phone call. Version one: silence for three days, then a finished bathroom in a gloss white you would never have picked. Version two: a photo of two tile samples at 2pm, "matte or gloss?", one word back, and the right bathroom. The difference isn't skill; it's that the second contractor knew which decision wasn't theirs to make.

item/tool/requestUserInput is version two, wired into the protocol. And its arrival mechanics should feel deeply familiar: it is a server-initiated JSON-RPC request, carrying an id, expecting a response, exactly like the approval requests of Part 7. The desk turns around again: the engine asks, your backend answers.

First, the honesty that belongs in the same breath as the feature. On a default thread, this request never fires. It sits behind a feature flag you saw in the thread/start fence above (features.default_mode_request_user_input), and when you set that flag, the engine sends a warning notification on every thread, verbatim from the wire:

Under-development features enabled: default_mode_request_user_input. Under-development features are incomplete and may behave unpredictably. To suppress this warning, set suppress_unstable_features_warning = true in $CODEX_HOME/config.toml.

Read that the way it's meant: the engine is telling you this surface may move under your product. Pagewright ships it anyway, clearly labeled, because the capability is worth teaching and the fallback is graceful (no flag, no questions, everything else works). Your production judgment may differ, and the honest wire message is exactly what you need to make it.

Here's the request itself on stdio, from the raw trace, with our response and the engine's acknowledgment:

Three shape facts to engrave, all verified live. questions is an array: one request can carry a whole intake form, several questions each with its own id, header, question text, isOther (free text allowed), isSecret (mask the input), and options as {label, description} pairs or null for pure free-text. The response wraps twice: {answers: {<question_id>: {answers: ["<label>"]}}}, a list per question id, then the whole sheet under one more answers key. Miss the double wrap and the request fails. And an answered question is a requirement, not a suggestion: the run that picked "Midnight Velvet" produced a page whose source contains that palette, named.

Bridging it costs almost nothing, and that fact is the design lesson of the whole series paying out. Part 2 built one seam for server-initiated requests; Part 7's approvals were its first two customers; this is the third, and the seam absorbs it without modification. Register a handler, park a Future, answer through an endpoint:

    # the agent asks the human a question and freezes until the answer
    # comes back down the same JSON-RPC pipe.
    async def question_handler(params: dict) -> dict:
        notify = client.queue_for(params["threadId"]).put
        return await questions.ask(params, notify)

    client.on_server_request("item/tool/requestUserInput", question_handler)

The new module, app/questions.py, is the foreman's inbox with a second drawer. One pending entry per request, holding the questions list and a Future as the mailbox:

class PendingQuestion:
    """One unanswered request (which may hold several questions). The
    Future is the mailbox: the JSON-RPC handler awaits it, the answer
    endpoint fills it with {question_id: [answers]}."""

    def __init__(self, thread_id: str, item_id: str, questions: list[dict]) -> None:
        self.id = uuid.uuid4().hex[:8]
        self.thread_id = thread_id
        self.item_id = item_id
        self.questions = questions
        self.timeout = timeout_seconds()
        self.expires_at_ms = int((time.time() + self.timeout) * 1000)
        self.future: asyncio.Future = asyncio.get_running_loop().create_future()


_registry: dict[str, PendingQuestion] = {}

ask() announces the question on the project's stream (a synthetic question_request event, the same trick approvals use, so the card lands in the transcript exactly where the turn paused, in every tab), then awaits the Future. The interesting half is what happens when nobody comes:

    try:
        answers = await asyncio.wait_for(pending.future, pending.timeout)
        reason = "user"
    except asyncio.TimeoutError:
        answers, reason = {}, "timeout"
    finally:
        _registry.pop(pending.id, None)
    await notify({"method": "question/resolved", "params": {
        "question_id": pending.id,
        "answers": answers,
        "reason": reason,
        "resolved_at_ms": int(time.time() * 1000),
    }})
    # The protocol's answer shape wraps each id's list one level deeper:
    # {answers: {id: {answers: [...]}}} (verified live).
    return {"answers": {qid: {"answers": ans} for qid, ans in answers.items()}}

The answering endpoint is the approvals decision endpoint's twin, with one extra courtesy: it validates that you're answering questions that were actually asked, so a stale or malformed sheet gets a clean 400 instead of a confused agent:

@app.post("/projects/{project_id}/questions/{question_id}/answer")
async def answer_question(project_id: str, question_id: str, req: AnswerRequest):
    entry = projects.get_project(project_id)
    if entry is None:
        raise HTTPException(status_code=404, detail="no such project")
    pending = questions.get(question_id)
    if pending is None or pending.thread_id != entry.get("thread_id"):
        raise HTTPException(status_code=404,
                            detail="no such pending question for this project")
    asked = {q.get("id") for q in pending.questions}
    if unknown := set(req.answers) - asked:
        raise HTTPException(status_code=400,
                            detail=f"answers for questions never asked: {sorted(unknown)}")
    if not questions.resolve(question_id, req.answers):
        raise HTTPException(status_code=404, detail="question already answered")
    return {"question_id": question_id, "answers": req.answers}

On the frontend, the card is the approval card's visual cousin (amber border while it hangs, outcome stamped in place when it resolves), and the block bookkeeping is the same patch-in-place pattern the transcript has used since Part 7:

  if (event.type === "question_request") {
    return [
      ...blocks,
      {
        type: "question",
        id: event.question_id,
        questions: event.questions,
        expiresAtMs: event.expires_at_ms,
      },
    ];
  }
  if (event.type === "question_resolved") {
    return blocks.map((b) =>
      b.type === "question" && b.id === event.question_id
        ? {
            ...b,
            resolved: {
              answers: event.answers,
              reason: event.reason,
              atMs: event.resolved_at_ms,
            },
          }
        : b,
    );
  }

Because the card is an event in the log, everything Part 9 bought applies without a line of new code: the question renders in every open tab, any tab can answer it, the resolution flips the card everywhere at once, and a tab opened after the answer replays the whole exchange in order. And note the small piece of protocol choreography that makes the card feel alive: the agent's tool call is frozen, but the notification pump is not (the dispatch task runs handlers concurrently, a decision made in Part 7 for approvals that this feature inherits), so the working timer ticks on while the question hangs.

One design question remains: when should the agent ask? That's a prompt-and-product decision, not a protocol one; the wire happily carries a question about anything. The working rule Pagewright inherits from the sibling series: ask when the decision is irreversible, expensive, or a matter of taste. Palette is taste. Brand voice is taste. "Should the nav be sticky" is not worth a frozen turn, and a model told to ask about everything will pepper the user into disabling the feature. Which is a warning the next comic takes literally.

Checkpoint. Right now you have: a builder that can freeze mid-job, hand you a structured form of one or many questions, honor the answer in the artifact it builds, and proceed honestly on its own judgment when the form times out. The bridge cost one module and one endpoint, because the Part 2 seam keeps paying compound interest: three server-request customers, zero seam changes.

The reasoning dial, finally on the composer

Since Part 3, two constants have ridden every turn without comment: how hard the model thinks, and how much of that thinking gets narrated. Time to admit they were dials all along. turn/start accepts effort and summary per turn; the legal efforts are whatever model/list advertises for the model (low, medium, high, xhigh for gpt-5.4-mini), and summary is auto, concise, or detailed. Per turn is the detail that makes this a product feature rather than a config file: one conversation can think cheap on a copy tweak at 2:03 and hard on a redesign at 2:04, no thread surgery, no new project.

# The reasoning dials, exactly as the engine advertises them: model/list
# names the efforts gpt-5.4-mini supports, and summary is the protocol
# enum minus "none" (Part 3 learned the hard way that "none" means the
# reasoning drawer stays empty forever).
EFFORTS = ("low", "medium", "high", "xhigh")
SUMMARIES = ("auto", "concise", "detailed")
DEFAULT_EFFORT = "medium"
DEFAULT_SUMMARY = "detailed"

Two protocol dials make a bad UI, though. "Effort xhigh, summary concise" is a sentence for people who read this series; your users are choosing between "quick tweak" and "take your time". So the composer folds both into one question, care level, four notches with opinionated pairings:

Quick drops the summary to concise deliberately: less thinking to narrate, less narration to read. Everything else keeps detailed, the setting the reasoning drawer has depended on since Part 3, and at Thorough and Max that drawer finally earns its keep: the summaries get long enough to actually watch the model weigh layouts against the brief. (House rule from the concepts page still applies: the narration is a summary the model writes about its own thinking, not a transcript; render it faint, never parse it.)

The send path threads the chosen pair onto every message, and the receipt threads it back (a non-default effort prints on the turn's receipt, so the transcript remembers which turns were expensive on purpose):

    // The dials ride every message. plan_first is one-shot: the toggle
    // arms ONE blueprint turn and disarms itself, so the natural
    // follow-up ("build it") goes out at the project's own mode without
    // the user having to remember the switch.
    const { effort, summary } = careDials(careLevel);
    const blueprint = planFirst && !working;
    try {
      const res = await fetch(`${API_BASE}/projects/${activeId}/chat`, {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          message: prompt,
          plan_first: blueprint,
          effort,
          summary,
        }),
      });
      if (!res.ok) throw new Error(`The server said ${res.status}.`);
      if (blueprint) setPlanFirst(false);

The obvious question, and it deserves numbers rather than adjectives: what does the dial actually buy?

Break it on purpose: the same brief at low and xhigh

This part's deliberate break isn't a traceback; it's a bill. We took the identical Beanline brief, two fresh projects, same model, same summary: "detailed", and turned only the dial: effort: "low" against effort: "xhigh". Real runs, receipts verbatim:

Read the rows separately, because they tell different stories. Total tokens moved by half, but reasoning tokens moved by twenty-five times: at low, 458 reasoning tokens is thinking as a formality, a model glancing at the brief and typing; at xhigh, 11,432 tokens is the model genuinely deliberating, and you can read the deliberation in the drawer. The qualitative gap matched the receipts: both runs completed the brief honestly (hero, four drinks, six stores, footer; nothing missing from either), but the xhigh page carried noticeably richer sectioning and typography detail, 38% more markup for the same content. The low page is correct and plainer. Neither run failed; that's what makes this a dial and not a quality switch.

So when do you pay for Max? The receipts suggest the rule: pay when judgment is the bottleneck, not typing. A hard brief with tensions to resolve, a redesign where structure is up for grabs, a review of someone else's mess: xhigh. Adding a paragraph, fixing a link, changing a color: low, and pocket the 93 seconds. And because the dial is per-message, the expensive thinking can be reserved for exactly the messages that need it, with the receipt printing effort xhigh so nobody wonders later why one turn cost triple. The dial is real, and so is the bill; the point of surfacing both is that the person holding the dial can finally see it.

The meter ritual, consultation edition

Every number a real run from building this part, per-turn deltas as Part 8 taught:

Read the first two rows together: the blueprint cost about a third of the build that followed it. That is the price of looking before swinging, and on a brief where the first swing would have gone the wrong way, it's the cheapest 62k tokens in the table.

Commit it, from the project root:

git add backend frontend
git commit -m "part 10: blueprint mode, living checklist, structured questions, reasoning dials"

Act III is open, and the builder has become a consultant: it proposes before demolition, publishes its estimate on the window, phones you about taste, and bills its thinking honestly. But every one of those courtesies still ends the same way, with the builder grading its own work. The builder plans and asks. But who checks the finished work? Next: the inspector.

The complete, tested code for this part lives in part-10-plans-questions in the companion repo. Code blocks with a GitHub icon link straight to the exact file; "View full file" shows the whole file in place with this section's lines highlighted.