Evals & benchmarks

Stories

• Harness-Delta Attribution: Dissecting Gains From Harness Evolution (HF Daily Papers)

  paper · Story page

  An evaluation protocol that attributes harness-evolution gains to benchmark overfitting, test-time compute, and a residual.

• How Kiro fixes itself, and how we keep the fixes honest (Kiro (AWS))

  blog post · Story page

  One false sentence in Kiro's tool descriptions reached the model 61,742 times across a few hundred sessions. Judging fixes is harder: one that passed more tasks on fewer tokens owed almost all its gain, once split by host, to a single run where the agent escaped a loop it usually gets stuck in.

• Inspect v0.3.283 (UK AISI Inspect releases)

  Story page

  Breaking change to dataset checkpoint fields

• UndoBench: Separating Task Competence from Recovery Capability in Tool-Using AI Agents (arXiv)

  paper · Story page

  Counterfactual paired trials under identical seeds pull fault recovery apart from ordinary planning competence. In the frozen lost-acknowledgment study nominal completion reached 83.54% while conditional recovery success fell to 46.72%, and naive retry produced duplicate external effects in 53.33% of trials.

• What agent traces can tell you without an LLM judge (Arize)

  blog post · Story page

  Proving agent failures from Phoenix spans and failing CI, no judge

• Your LLM Judge Is a Confident Liar: Building Better Verifiers (Browserbase (AI Engineer))

  talk · Story page

  The official judge said 74%, a better verifier said 38%

• DelegationBench: Measuring When AI Agents Should Ask Before Acting (arXiv)

  paper · Story page

  Rewording the same question shifts action rates by up to 52.5 percentage points

• ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time (arXiv)

  paper · Story page

  ArrivalBench re-runs the pipeline an agent leaves behind against late, duplicated, out-of-order and retried records, then requires the final table to match a full recomputation of the log. Single-execution grading certified 86 to 100% of the pipelines eleven models produced; replaying the same artifacts found 7.0 to 79.2% of those silently wrong.

• ReviewBench: An open benchmark for AI code review (GitHub Blog)

  blog post · Story page

  GitHub's open benchmark for code review agents

• ToolBench Rescan: 90,000 MCP Servers Scored (Arcade.dev)

  Story page

  Six months on, and 0.28% of servers earn an A

• Language Models Are "Insecure" Reporters (arXiv)

  paper · Story page

  Given experiment logs with a planted negative result that weakens its own method, GPT-5.5 flagged it in 2 of 200 reports, and in 190 of 200 once "Be honest in your response" was added.

• Claude’s new auto eval tool (Hamel Husain)

  blog post · Story page

  Husain and Isaac Flath livestreamed Claude Code's new build_eval and hill-climb commands over traces from an apartment leasing assistant. His sharpest complaint is ordering: it asked them to pick a failure before they'd read any conversations.

• What TLA+ can and can't check (Hillel Wayne (Computer Things))

  newsletter item · Story page

  The properties TLA+ can't even express

• The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge (arXiv)

  paper · Story page

  KNOWS makes a browser agent research something and then hand back a finished document, presentation or spreadsheet, graded by deterministic checks mixed with LLM judgments. Frontier agents score moderately on partial success, and the best performer fully succeeds on fewer than 3% of tasks.

• How to build frontier RL browser tasks (Browserbase and HUD)

  blog post · Story page

  Prompt, environment, grader, and QA agents reading traces for reward hacking

• Claude Sonnet 5.5 reaches #2 on the Artificial Analysis Intelligence Index (Artificial Analysis)

  blog post · Story page

  Second at max effort, about 50% higher cost per task

• How We Built an Agent That Improves Itself (Zubin Aysola (Weights & Biases))

  talk · via AI Engineer (talks) · Story page

  Weights & Biases turns a production trace into an offline eval task, finds the bug, writes the fix and benchmarks the new agent against the one in production. The research and production agents are kept byte-for-byte identical, and the suite runs 886 tasks including simulated multi-turn users.

• From Answers to Audit Opinions: Cost-Aware Expert Verification of Financial Artifacts from Tool-Using LLM Agents (Semantic Scholar)

  paper · Story page

  Audit-opinion labels and a certificate for what an agent produced

• What Makes a Terminal-Bench Task Hard? Separating Genuine Hardness from Fake-Hardness on an Adjudicated Agentic Corpus (arXiv)

  paper · Story page

  Only 78 of the 125 all-fail tasks in a frozen Terminal-Bench 3 and Frontier-Bench 0.1 record survive as certified-unsolved candidates. The rest involve broken oracles, infrastructure failures, verifier bypasses, or tasks with insufficient evidence of solvability.

• Opus 5.5, GPT-6 Sol and Grok 4.7 on Vending-Bench (Andon Labs Blog)

  Story page

  GPT-6 Sol averaged $14,428; Opus 5.5 made less than Opus 5

• Passes Alone, Fails Together: Benchmarking Semantic Coordination in Parallel LLM-Agent Development (arXiv)

  paper · Story page

  Two agents each write a patch that passes alone, then the pair breaks when combined. Interference hit 97% of runs on constructed tasks using 12 real Django helpers, and one of 834 runs on mined pull-request pairs; the authors say the constructed rate estimates nothing about practice.

• Opus 5.5 vs Opus 5 vs Fable 5.1: Coding Benchmark Results (Snorkel AI)

  blog post · Story page

  Pass@1 is flat across three generations on one expert-created terminal-bench style task set: 61.5% for Fable 5.1, 60.7% for Opus 5, and 60.7% for Opus 5.5. Pass@5 separates them, and Opus 5.5's 76.7% sits below Opus 5's 79.3%.

• SWE-Bench Pro V2: A Cleaner, Harder-to-Game Leaderboard (Scale Labs (SEAL))

  Story page

  A refreshed public split, network-locked evaluation protocol

• SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs? (arXiv)

  paper · Story page

  Benchproofer turns 500 real SWE-bench Verified issues into formally verified tasks, admitting an instance only once mechanical and adversarial gates agree on its specification. Across two frontier models, a quarter to a half of test-passing patches admit counterexamples.

• What Stops a Small Language Model From Driving a Database Agent (arXiv)

  paper · Story page

  Eleven days driving an open-source SQL client's agent mode with 39 local open-weight models produced 8,199 runs. Of 2,100 model-attributed losses, 1,590, or 75.7%, came from runs that had already invoked a tool, so the bottleneck sits after the call.

• When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success (arXiv)

  paper · Story page

  Supervised fine-tuning raised next-turn success for every Qwen3 and Gemma 3 model tested against gold histories, and none of the four then cleared holistic workflow evaluation, with strict trajectory completion topping out at 10.4%. The authors want the layers reported separately.

• Autopilot: engineering an agentic quality loop for support automation (Coinbase Engineering)

  blog post · Story page

  Every Coinbase support procedure is a versioned decision tree, and Autopilot runs agents that author, test, red-team, score and analyze it behind contracts, fixtures, CI gates and human sign-off on writes. The post says this makes judgment cheaper and more repeatable, not optional.

• Benchmarking Grok 4.7 (Artificial Analysis)

  blog post · via xAI News · Story page

  Independent scores, agentic and coding and long-context

• AI Evals: Everything You Need to Know (Hamel Husain)

  blog post · Story page

  An FAQ from teaching 700+ engineers and PMs

• Are LLM Performance Benchmarks Reliable? (Ashok Chandrasekar and Jason Kramberger (Google))

  talk · Story page

  Ask a harness for 200 queries a second and it may deliver 38, then print the results as though it ran 200. The global interpreter lock caps a single-process client, one shared 20% throughput win ran at temperature zero, and Inference Perf logs scheduled against actual send times.

• Can Jev Be a Better Agent Evaluator? (LangChain)

  blog post · Story page

  Jev tested against LLM judges as an agent evaluator

• Locating Hidden Failures Makes Long-Horizon Agents More Reliable (arXiv)

  paper · Story page

  Hand annotation of 2,518 agent trajectories sorts 6,967 mistakes into 78 failure types. Runs scored as solved still delete data, corrupt systems or fabricate success, and six frontier judges struggle to locate where a run went wrong.

• Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations (arXiv)

  paper · Story page

  Difference-of-means vectors read off model internals catch reward hacking about as well as LLM monitors, at almost no cost. Getting there meant measuring the hacking: GLM 5.2 hacks in 57.2% of DeepSWE rollouts and 73% of SWE-bench rollouts.

• Introducing Exa Snapshot, A New Way to Search the Past (Exa)

  Story page

  Exa Snapshot returns the web as it stood on a date you pass, from over 400 billion page snapshots spanning two decades. Set snapshotAsOf before a task's answer was published and Snapshot search won't return that solution to an agent in RL training.

• Run Terminal-Bench and other Harbor evals on Vercel Sandbox (Vercel)

  blog post · Story page

  One Firecracker microVM per Harbor trial, firewall outside the VM

• ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software (arXiv)

  paper · Story page

  Screenshot-only agents scored against a live ERP system's database

• HarnessTax: How Much Does the Harness Matter for Coding Agents? (Hacker News)

  HN thread · 143 points on HN · Story page

  How much of a coding agent's score is the harness

• How to find and debug agent failures your evals are missing (Arize)

  blog post · Story page

  Reading production traces for the trajectory failures nobody named

• Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead (arXiv)

  paper · Story page

  Across 254 submissions, read without running a model, the two leading Verified entries each resolve 396 of 500 instances and exact paired McNemar tests separate none of the 29 adjacent pairs in the top thirty. Within-model scaffold ranges reach 29.8 points against that group's 8.8-point spread.

• Fabrication After Tool Failure: Tool-Augmented Agents Assert Values Their Tools Did Not Return (arXiv (via papers.cool))

  paper · Story page

  Whether a model invents a value after a tool fails turns on signalling: 0.0% dishonesty when the tool returned status:error, 45.3% when it returned status:ok with an unusable value. None of the nine frameworks audited says what to do when a tool fails.

• When Tools Get in the Way: The Effect of Unnecessary Tool Availability on LLM Answering (arXiv (via papers.cool))

  paper · Story page

  Across 3,000 baseline trials the pooled answer rate is 98.2%; add a related but unnecessary tool and it falls to 63.5%. The drop holds even when the tool is rarely called.

• GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents (arXiv (via papers.cool))

  paper · Story page

  When the simulate-and-judge release gate stops tracking real task success

• The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean (arXiv)

  paper · Story page

  Two ways an agent benchmark can measure its own pipeline instead of the model: a fixed scaffold makes the execution-critical decisions, and the scorer may reward output shape rather than correctness. Handing those decisions to the model, scoring against seeded ground truth and reporting worst-case metrics turned ComtradeBench's nearly flat leaderboard into a spread of reliability.

• 3DHarnessBench: Probing Agentic 3D-to-Code Capabilities of Frontier Vision-Language Models (arXiv)

  paper · Story page

  Blender-code reconstruction benchmark across four levels of tool access

• Automated agent evaluation with Amazon Bedrock AgentCore and GitHub Actions (AWS ML Blog)

  blog post · Story page

  A GitHub Actions pipeline deploys an agent and an OAuth-protected MCP server to AgentCore runtime, fires test prompts, scores the answers and blocks the pull request when behaviour regresses. Vendor walkthrough, but the shape, agent evals as a merge gate, transfers.

• $τ^τ$-Bench: An Environment for End-To-End, Realistic Agent Construction (arXiv)

  paper · Story page

  Building the customer-service agent is the benchmark task

• Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation (arXiv)

  paper · Story page

  More than 80 agent benchmarks behind one adapter layer

• NInfer vs llama.cpp vs vLLM: quality + speed comparison for Qwen3.8-27B NVFP4 on RTX 5090 (r/LocalLLaMA)

  Reddit thread · Story page

  Three engines serving one Qwen3.8-27B on one RTX 5090

• Qwen3.8 Flash Next - Templates Comparison (r/LocalLLaMA)

  Reddit thread · Story page

  Three chat templates, one model, three SWE-bench Verified rates

• GPT-6 Astra (Simon Willison)

  blog post · Story page

  Simon Willison rounds up GPT-6 Astra's rollout, pricing, and OpenAI's self-reported benchmarks, including the gap between OpenAI's own harness and the default one on ARC-AGI 3.

• How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making (arXiv)

  paper · Story page

  Across nine models, agent task success follows a geometric law set by one per-step reliability parameter, which rises with model scale but saturates well below 1. On the tool-use task, every model tested fell from near-perfect to near zero within sixteen dependent steps, which the authors argue is why benchmark optimism doesn't survive production horizons.

• LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails (arXiv)

  paper · Story page

  After months of running autonomous prompt-optimization loops in production, the authors catalog eleven ways the evaluation signal failed, including agents that hit a 100% pass rate by reading cached answer keys, concealing 68% true capability. Their fix demotes the LLM judge to advisor and gates every change behind deterministic checks it can't override.

• Introducing Factory Benchmarks (Warp)

  blog post · Story page

  Early-access agent benchmarks on your own code, inside Warp Factories

• Qwen's E-Commerce Bench Exposes How Badly AI Agents Fail at Running a Business (AlphaSignal)

  newsletter item · Story page

  Qwen's 365-day marketplace sim, where no model wins everything

• BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks (arXiv)

  paper · Story page

  Planted optional shortcuts that inflate public test scores

• RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests (arXiv (via papers.cool))

  arXiv, 4 pp · Story page

  RealSWE evaluates coding agents on task variants that keep the task and gold patch fixed while varying only the request's information content and writing style.

• LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis (arXiv)

  paper · Story page

  LongDS builds 68 long-horizon data-analysis tasks from real Kaggle notebooks, spanning 2,225 turns. The best of five models reaches 48.45%, accuracy falls nearly 47 points from early to late turns, and extra steps don't necessarily help, suggesting the bottleneck is keeping analytical state correct.

• What survives honest evaluation? Leakage-safe, search-aware assessment of LLM-driven trading strategy discovery (Semantic Scholar)

  paper · Story page

  leakage-safe, search-aware evaluation of LLM trading agents

• SOTA Generative Media Panel (Dumitru Erhan, Shane Gu & Nicole Brichtova (Google DeepMind))

  talk · Story page

  The panel regenerated real videos from their captions and human evaluators largely preferred the generated version: sharper and more saturated, Dumitru Erhan deflates, not more realistic. Their image model also quietly added wedding rings nobody caught internally, a warning for anyone scoring agents on preference signals.

• This finance-model benchmark card is more useful for what it discloses than for who "wins" (r/LocalLLaMA)

  Reddit thread · Story page

  The Ling-3.0-flash-Fin card discloses what got scored: a common ReAct scaffold with web search and Python, Claude Code 2.1.173 driving LibreOffice 25.8.7, turn limits, timeouts, and a GPT-5 judge. The unit under test is the agent system, not the checkpoint, and the post reads as a checklist for any agent benchmark.

• Artificial Analysis Catches AI Coding Agents Cheating Their Way to Top Scores (AlphaSignal)

  newsletter item · Story page

  Terminal-Bench runs that fetch solutions online now score zero

• How Signal found two hidden retry loops in our production agent Alyx (Arize)

  blog post · Story page

  A 43-call retry that looked like healthy tool activity

• Signal or Noise? A Benchmark Study of Agent Skills in Web Development (arXiv)

  paper · Story page

  Across 31 public WebDev Skills, 50 projects and 1,000 tasks on four models, injecting the target Skill cut mean Pass@2 by 1.3% to 4.2% and raised token cost by 72% to 394%, with gains in only 17% to 36% of Skill-project pairs. On some models an equally long irrelevant Skill reproduced most of the loss.

• There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items (arXiv)

  paper · Story page

  One model scores 31 to 89 percent depending only on harness config

• Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills (arXiv (via papers.cool))

  paper · Story page

  ACES answers what a static skill scan can't: does the package help a live agent? It runs paired trials with and without each skill under the same model, sandbox, tasks, and scorer, then reports the measured lift, evaluated on 145 real skills.

• Sandboxed agent evals with Harbor (Braintrust)

  blog post · Story page

  Sandboxed Docker eval tasks with verifiers, now synced to Braintrust

• How to Evaluate Live & Voice Agents in ADK (Google Developers Blog)

  blog post · Story page

  Automated evals for live voice agents, with simulated audio users

• Introducing Multi-turn Evals for Mastra Agents (Mastra)

  Story page

  Multi-turn conversation evals with deterministic gates and judge scorers

• Qwen3.8-27B: One Week Later: The r/LocalLLaMA + r/LocalLLM Verdict (r/LocalLLaMA)

  Reddit thread · Story page

  A week of community verdicts on Qwen 3.8 27B, disagreements kept

• Metrics That Write Themselves: Evolving an Evaluator from Its Own Blind Spots (arXiv (via papers.cool))

  paper · Story page

  an evaluator that authors defect detectors from its own blind spots

• Your Agent Evolved. Your Evals Didn't. (Ameya Bhatawdekar (Braintrust))

  talk · Story page

  why eval design must move with agent architectures

• Shipping AI to a Million Patients Without an A/B Test (Jared Joselowitz (Ufonia))

  talk · Story page

  Ufonia's regulated clinical voice agent has made roughly 200,000 calls, and you can't A/B test on patients, so experimentation moves into simulation: one model plays the patient from clinician-written hazards, a second judges every dialogue, and both roles were validated rather than assumed.

• StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows (arXiv (via papers.cool))

  paper · Story page

  Agent benchmark built from market-validated startup workflows

• Open-sourcing evals for open-weight agents (Cline)

  blog post · Story page

  Cline says its five most-used models are all open-weight, so it's publishing the evals it uses to judge them on Terminal Bench, tied to model behaviour and token efficiency inside its own harness rather than a bare benchmark score.

• Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation (arXiv)

  paper · Story page

  Coding-agent pass@k sets n to unit tests, not independent rollouts

• Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents (Semantic Scholar)

  paper · Story page

  A differential method pins a failed or costlier run on the skill it loaded by comparing it with a no-skill or matched-skill run of the same task. On SkillsBench and SWE-Skills-Bench that yields 307 skill-induced failures, 125 functional and 182 efficiency.

• Retry, Switch, or Abstain? Learning Strategy-Aware Tool-Use Policies via Controlled Error Injection (Semantic Scholar)

  paper · Story page

  BENCH2ROBUST injects transient, persistent and silent tool failures into benchmarks that never fail, so retry, switch and stop become trainable choices. Across 7 models from 4 families the recovery gap is near universal; Bayesian Tool Memory adds up to 16.8 points without retraining.

• How many of your agent's calls actually need a frontier model? (LangChain)

  blog post · Story page

  Routing cut cost 74% for six accuracy points across 145 tasks.

• How Uber evaluates AI agents at production scale (Arize)

  blog post · Story page

  A background comment about pizza exposed a failure Uber's offline agent evaluations had missed; the post lays out what production evaluation needs instead.

• Computer Use at the Edge of the Statistical Precipice (AI Engineer (talks): Pierluca D'Oro, Programma Labs)

  talk · Story page

  A blind replay of one recorded trajectory per task matches or beats the frontier model it was copied from on deterministic benchmarks like OSWorld.

• SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries (arXiv)

  paper · Story page

  SteerBench-Work tests the pre-commit choice before consequential tool actions: proceed, or hold for review, across 106 incident-anchored workplace scenarios.

• METR Raises $71M to Independently Stress-Test the World's Most Powerful AI (AlphaSignal)

  newsletter item · Story page

  $71M raised in six months for independent safety evaluations.

• Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations (arXiv)

  paper · Story page

  A generalizability-theory analysis of three open agent benchmarks finds leaderboard scores dominated by agent-by-task interaction rather than by which agent you picked.

• Artificial Analysis' Optima Lets Any Team Build Custom AI Benchmarks (AlphaSignal)

  newsletter item · Story page

  Artificial Analysis launched Optima, which builds custom benchmarks from your own data, agent traces, or a plain-language description, tracking cost and speed alongside quality.

• InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk (arXiv)

  paper · Story page

  Realistic infrastructure tasks with fine-grained risk scoring

• AutoWorldModel-Bench: A State-Centric Benchmark for Automated World-Model Research (arXiv)

  paper · Story page

  A benchmark for agents doing autonomous world-model research

• Behavior specs, an open standard for supervising long-horizon agents (Braintrust)

  blog post · Story page

  Braintrust and Basis release an open standard for judging how a long-horizon agent behaves across a trajectory, not only its final result: in tax, a correct return doesn't tell you it was reached the right way. Each score runs the full trajectory, so evals are expensive and iteration slow.

• DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments? (arXiv)

  paper · Story page

  275 end-to-end data-science tasks in real computer environments

• Beyond Static Intelligence: Evaluating Continual Learning (Parth Asawa (UC Berkeley))

  talk · Story page

  Continual learning measured as gain over the same system reset

• AndroidReality: How Far Are Mobile Agents from the Real World? (arXiv)

  paper · Story page

  AndroidReality injects realistic state, transition, and action perturbations into AndroidWorld to measure how far mobile agents fall from their clean-benchmark numbers. It finds substantial failure gaps and a training-free recovery step that helps in both settings.

• When LLM Agents Negotiate: Private Information and Dynamic Bargaining in Supply Chains (arXiv)

  paper · Story page

  Nine LLMs negotiated 9,840 buyer-seller contracts against a game-theoretic benchmark. Agents captured 95.4% of first-best surplus, but slow bargaining eroded 21–34% of it, and baseline models accepted individually irrational contracts in 19.2% of cases.

• UCLA Finds AI Reward Hack Monitors Collapse to 28% on Real Cheating (AlphaSignal)

  newsletter item · Story page

  Reward-hacking monitors trained on synthetic examples fall to 28% on real model cheating; the synthetic data doesn't reflect how models actually exploit RL rewards. If a monitor is your safety layer, it may be watching for the wrong thing.

• Benchmarking Coding Agents on New vs Legacy Codebases (Denys Linkov (Wisedocs))

  talk · Story page

  Linkov audits a six-month, ten-repository medical-claims refactor to ask whether agents could have done it. The same task took three hours and ten major mistakes with o3, while Opus 4.8 essentially got it in one pass; handed the whole job, GPT 5.5 declared it done in about ten minutes with the actual models missing.

• AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection (arXiv)

  paper · Story page

  Chaos engineering for agent stacks via LLM API fault injection

• DeepSeek V4 Flash 0731 hits 82.7% on Terminal-Bench 2.1 in an independent public-harness run (445 trials) (r/LocalLLaMA)

  Reddit thread · Story page

  An independent public-harness run matched DeepSeek's reported 82.7% Terminal-Bench 2.1 score for V4 Flash 0731 over 445 trials.

• Guide, Verify, Solve (Anirban Chatterjee (Sonar))

  talk · Story page

  Chatterjee argues AI-written code leaves verification debt that human review alone can't reliably contain: a Wharton study he cites suggests reviewers followed AI advice nearly 80% of the time even when it was instructed to lie confidently. His fix is zero-trust, multilayer verification that checks generated code by methods independent of the model that wrote it.

Issues that covered it