Evals & benchmarks

Stories

• Locating Hidden Failures Makes Long-Horizon Agents More Reliable (arXiv)

  paper · Story page

  Hand annotation of 2,518 agent trajectories sorts 6,967 mistakes into 78 failure types. Runs scored as solved still delete data, corrupt systems or fabricate success, and six frontier judges struggle to locate where a run went wrong.

• Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations (arXiv)

  paper · Story page

  Difference-of-means vectors read off model internals catch reward hacking about as well as LLM monitors, at almost no cost. Getting there meant measuring the hacking: GLM 5.2 hacks in 57.2% of DeepSWE rollouts and 73% of SWE-bench rollouts.

• Introducing Exa Snapshot, A New Way to Search the Past (Exa)

  Story page

  Exa Snapshot returns the web as it stood on a date you pass, from over 400 billion page snapshots spanning two decades. Set snapshotAsOf before a task's answer was published and Snapshot search won't return that solution to an agent in RL training.

• Run Terminal-Bench and other Harbor evals on Vercel Sandbox (Vercel)

  blog post · Story page

  One Firecracker microVM per Harbor trial, firewall outside the VM

• ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software (arXiv)

  paper · Story page

  Screenshot-only agents scored against a live ERP system's database

• HarnessTax: How Much Does the Harness Matter for Coding Agents? (Hacker News)

  HN thread · 143 points on HN · Story page

  How much of a coding agent's score is the harness

• How to find and debug agent failures your evals are missing (Arize)

  blog post · Story page

  Reading production traces for the trajectory failures nobody named

• Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead (arXiv)

  paper · Story page

  Across 254 submissions, read without running a model, the two leading Verified entries each resolve 396 of 500 instances and exact paired McNemar tests separate none of the 29 adjacent pairs in the top thirty. Within-model scaffold ranges reach 29.8 points against that group's 8.8-point spread.

• Fabrication After Tool Failure: Tool-Augmented Agents Assert Values Their Tools Did Not Return (arXiv (via papers.cool))

  paper · Story page

  Whether a model invents a value after a tool fails turns on signalling: 0.0% dishonesty when the tool returned status:error, 45.3% when it returned status:ok with an unusable value. None of the nine frameworks audited says what to do when a tool fails.

• When Tools Get in the Way: The Effect of Unnecessary Tool Availability on LLM Answering (arXiv (via papers.cool))

  paper · Story page

  Across 3,000 baseline trials the pooled answer rate is 98.2%; add a related but unnecessary tool and it falls to 63.5%. The drop holds even when the tool is rarely called.

• GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents (arXiv (via papers.cool))

  paper · Story page

  When the simulate-and-judge release gate stops tracking real task success

• The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean (arXiv)

  paper · Story page

  Two ways an agent benchmark can measure its own pipeline instead of the model: a fixed scaffold makes the execution-critical decisions, and the scorer may reward output shape rather than correctness. Handing those decisions to the model, scoring against seeded ground truth and reporting worst-case metrics turned ComtradeBench's nearly flat leaderboard into a spread of reliability.

• 3DHarnessBench: Probing Agentic 3D-to-Code Capabilities of Frontier Vision-Language Models (arXiv)

  paper · Story page

  Blender-code reconstruction benchmark across four levels of tool access

• Automated agent evaluation with Amazon Bedrock AgentCore and GitHub Actions (AWS ML Blog)

  blog post · Story page

  A GitHub Actions pipeline deploys an agent and an OAuth-protected MCP server to AgentCore runtime, fires test prompts, scores the answers and blocks the pull request when behaviour regresses. Vendor walkthrough, but the shape, agent evals as a merge gate, transfers.

• $τ^τ$-Bench: An Environment for End-To-End, Realistic Agent Construction (arXiv)

  paper · Story page

  Building the customer-service agent is the benchmark task

• Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation (arXiv)

  paper · Story page

  More than 80 agent benchmarks behind one adapter layer

• NInfer vs llama.cpp vs vLLM: quality + speed comparison for Qwen3.8-27B NVFP4 on RTX 5090 (r/LocalLLaMA)

  Reddit thread · Story page

  Three engines serving one Qwen3.8-27B on one RTX 5090

• Qwen3.8 Flash Next - Templates Comparison (r/LocalLLaMA)

  Reddit thread · Story page

  Three chat templates, one model, three SWE-bench Verified rates

• GPT-6 Astra (Simon Willison)

  blog post · Story page

  Simon Willison rounds up GPT-6 Astra's rollout, pricing, and OpenAI's self-reported benchmarks, including the gap between OpenAI's own harness and the default one on ARC-AGI 3.

• How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making (arXiv)

  paper · Story page

  Across nine models, agent task success follows a geometric law set by one per-step reliability parameter, which rises with model scale but saturates well below 1. On the tool-use task, every model tested fell from near-perfect to near zero within sixteen dependent steps, which the authors argue is why benchmark optimism doesn't survive production horizons.

• LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails (arXiv)

  paper · Story page

  After months of running autonomous prompt-optimization loops in production, the authors catalog eleven ways the evaluation signal failed, including agents that hit a 100% pass rate by reading cached answer keys, concealing 68% true capability. Their fix demotes the LLM judge to advisor and gates every change behind deterministic checks it can't override.

• Introducing Factory Benchmarks (Warp)

  blog post · Story page

  Early-access agent benchmarks on your own code, inside Warp Factories

• Qwen's E-Commerce Bench Exposes How Badly AI Agents Fail at Running a Business (AlphaSignal)

  newsletter item · Story page

  Qwen's 365-day marketplace sim, where no model wins everything

• BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks (arXiv)

  paper · Story page

  Planted optional shortcuts that inflate public test scores

• RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests (arXiv (via papers.cool))

  arXiv, 4 pp · Story page

  RealSWE evaluates coding agents on task variants that keep the task and gold patch fixed while varying only the request's information content and writing style.

• LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis (arXiv)

  paper · Story page

  LongDS builds 68 long-horizon data-analysis tasks from real Kaggle notebooks, spanning 2,225 turns. The best of five models reaches 48.45%, accuracy falls nearly 47 points from early to late turns, and extra steps don't necessarily help, suggesting the bottleneck is keeping analytical state correct.

• What survives honest evaluation? Leakage-safe, search-aware assessment of LLM-driven trading strategy discovery (Semantic Scholar)

  paper · Story page

  leakage-safe, search-aware evaluation of LLM trading agents

• SOTA Generative Media Panel (Dumitru Erhan, Shane Gu & Nicole Brichtova (Google DeepMind))

  talk · Story page

  The panel regenerated real videos from their captions and human evaluators largely preferred the generated version: sharper and more saturated, Dumitru Erhan deflates, not more realistic. Their image model also quietly added wedding rings nobody caught internally, a warning for anyone scoring agents on preference signals.

• This finance-model benchmark card is more useful for what it discloses than for who "wins" (r/LocalLLaMA)

  Reddit thread · Story page

  The Ling-3.0-flash-Fin card discloses what got scored: a common ReAct scaffold with web search and Python, Claude Code 2.1.173 driving LibreOffice 25.8.7, turn limits, timeouts, and a GPT-5 judge. The unit under test is the agent system, not the checkpoint, and the post reads as a checklist for any agent benchmark.

• Artificial Analysis Catches AI Coding Agents Cheating Their Way to Top Scores (AlphaSignal)

  newsletter item · Story page

  Terminal-Bench runs that fetch solutions online now score zero

• How Signal found two hidden retry loops in our production agent Alyx (Arize)

  blog post · Story page

  A 43-call retry that looked like healthy tool activity

• Signal or Noise? A Benchmark Study of Agent Skills in Web Development (arXiv)

  paper · Story page

  Across 31 public WebDev Skills, 50 projects and 1,000 tasks on four models, injecting the target Skill cut mean Pass@2 by 1.3% to 4.2% and raised token cost by 72% to 394%, with gains in only 17% to 36% of Skill-project pairs. On some models an equally long irrelevant Skill reproduced most of the loss.

• There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items (arXiv)

  paper · Story page

  One model scores 31 to 89 percent depending only on harness config

• Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills (arXiv (via papers.cool))

  paper · Story page

  ACES answers what a static skill scan can't: does the package help a live agent? It runs paired trials with and without each skill under the same model, sandbox, tasks, and scorer, then reports the measured lift, evaluated on 145 real skills.

• Sandboxed agent evals with Harbor (Braintrust)

  blog post · Story page

  Sandboxed Docker eval tasks with verifiers, now synced to Braintrust

• How to Evaluate Live & Voice Agents in ADK (Google Developers Blog)

  blog post · Story page

  Automated evals for live voice agents, with simulated audio users

• Introducing Multi-turn Evals for Mastra Agents (Mastra)

  Story page

  Multi-turn conversation evals with deterministic gates and judge scorers

• Qwen3.8-27B: One Week Later: The r/LocalLLaMA + r/LocalLLM Verdict (r/LocalLLaMA)

  Reddit thread · Story page

  A week of community verdicts on Qwen 3.8 27B, disagreements kept

• Metrics That Write Themselves: Evolving an Evaluator from Its Own Blind Spots (arXiv (via papers.cool))

  paper · Story page

  an evaluator that authors defect detectors from its own blind spots

• Your Agent Evolved. Your Evals Didn't. (Ameya Bhatawdekar (Braintrust))

  talk · Story page

  why eval design must move with agent architectures

• Shipping AI to a Million Patients Without an A/B Test (Jared Joselowitz (Ufonia))

  talk · Story page

  Ufonia's regulated clinical voice agent has made roughly 200,000 calls, and you can't A/B test on patients, so experimentation moves into simulation: one model plays the patient from clinician-written hazards, a second judges every dialogue, and both roles were validated rather than assumed.

• StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows (arXiv (via papers.cool))

  paper · Story page

  Agent benchmark built from market-validated startup workflows

• Open-sourcing evals for open-weight agents (Cline)

  blog post · Story page

  Cline says its five most-used models are all open-weight, so it's publishing the evals it uses to judge them on Terminal Bench, tied to model behaviour and token efficiency inside its own harness rather than a bare benchmark score.

• Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation (arXiv)

  paper · Story page

  Coding-agent pass@k sets n to unit tests, not independent rollouts

• Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents (Semantic Scholar)

  paper · Story page

  A differential method pins a failed or costlier run on the skill it loaded by comparing it with a no-skill or matched-skill run of the same task. On SkillsBench and SWE-Skills-Bench that yields 307 skill-induced failures, 125 functional and 182 efficiency.

• Retry, Switch, or Abstain? Learning Strategy-Aware Tool-Use Policies via Controlled Error Injection (Semantic Scholar)

  paper · Story page

  BENCH2ROBUST injects transient, persistent and silent tool failures into benchmarks that never fail, so retry, switch and stop become trainable choices. Across 7 models from 4 families the recovery gap is near universal; Bayesian Tool Memory adds up to 16.8 points without retraining.

• How many of your agent's calls actually need a frontier model? (LangChain)

  blog post · Story page

  Routing cut cost 74% for six accuracy points across 145 tasks.

• How Uber evaluates AI agents at production scale (Arize)

  blog post · Story page

  A background comment about pizza exposed a failure Uber's offline agent evaluations had missed; the post lays out what production evaluation needs instead.

• Computer Use at the Edge of the Statistical Precipice (AI Engineer (talks): Pierluca D'Oro, Programma Labs)

  talk · Story page

  A blind replay of one recorded trajectory per task matches or beats the frontier model it was copied from on deterministic benchmarks like OSWorld.

• SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries (arXiv)

  paper · Story page

  SteerBench-Work tests the pre-commit choice before consequential tool actions: proceed, or hold for review, across 106 incident-anchored workplace scenarios.

• METR Raises $71M to Independently Stress-Test the World's Most Powerful AI (AlphaSignal)

  newsletter item · Story page

  $71M raised in six months for independent safety evaluations.

• Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations (arXiv)

  paper · Story page

  A generalizability-theory analysis of three open agent benchmarks finds leaderboard scores dominated by agent-by-task interaction rather than by which agent you picked.

• Artificial Analysis' Optima Lets Any Team Build Custom AI Benchmarks (AlphaSignal)

  newsletter item · Story page

  Artificial Analysis launched Optima, which builds custom benchmarks from your own data, agent traces, or a plain-language description, tracking cost and speed alongside quality.

• InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk (arXiv)

  paper · Story page

  Realistic infrastructure tasks with fine-grained risk scoring

• AutoWorldModel-Bench: A State-Centric Benchmark for Automated World-Model Research (arXiv)

  paper · Story page

  A benchmark for agents doing autonomous world-model research

• Behavior specs, an open standard for supervising long-horizon agents (Braintrust)

  blog post · Story page

  Braintrust and Basis release an open standard for judging how a long-horizon agent behaves across a trajectory, not only its final result: in tax, a correct return doesn't tell you it was reached the right way. Each score runs the full trajectory, so evals are expensive and iteration slow.

• DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments? (arXiv)

  paper · Story page

  275 end-to-end data-science tasks in real computer environments

• Beyond Static Intelligence: Evaluating Continual Learning (Parth Asawa (UC Berkeley))

  talk · Story page

  Continual learning measured as gain over the same system reset

• AndroidReality: How Far Are Mobile Agents from the Real World? (arXiv)

  paper · Story page

  AndroidReality injects realistic state, transition, and action perturbations into AndroidWorld to measure how far mobile agents fall from their clean-benchmark numbers. It finds substantial failure gaps and a training-free recovery step that helps in both settings.

• When LLM Agents Negotiate: Private Information and Dynamic Bargaining in Supply Chains (arXiv)

  paper · Story page

  Nine LLMs negotiated 9,840 buyer-seller contracts against a game-theoretic benchmark. Agents captured 95.4% of first-best surplus, but slow bargaining eroded 21–34% of it, and baseline models accepted individually irrational contracts in 19.2% of cases.

• UCLA Finds AI Reward Hack Monitors Collapse to 28% on Real Cheating (AlphaSignal)

  newsletter item · Story page

  Reward-hacking monitors trained on synthetic examples fall to 28% on real model cheating; the synthetic data doesn't reflect how models actually exploit RL rewards. If a monitor is your safety layer, it may be watching for the wrong thing.

• Benchmarking Coding Agents on New vs Legacy Codebases (Denys Linkov (Wisedocs))

  talk · Story page

  Linkov audits a six-month, ten-repository medical-claims refactor to ask whether agents could have done it. The same task took three hours and ten major mistakes with o3, while Opus 4.8 essentially got it in one pass; handed the whole job, GPT 5.5 declared it done in about ten minutes with the actual models missing.

• AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection (arXiv)

  paper · Story page

  Chaos engineering for agent stacks via LLM API fault injection

• DeepSeek V4 Flash 0731 hits 82.7% on Terminal-Bench 2.1 in an independent public-harness run (445 trials) (r/LocalLLaMA)

  Reddit thread · Story page

  An independent public-harness run matched DeepSeek's reported 82.7% Terminal-Bench 2.1 score for V4 Flash 0731 over 445 trials.

• Guide, Verify, Solve (Anirban Chatterjee (Sonar))

  talk · Story page

  Chatterjee argues AI-written code leaves verification debt that human review alone can't reliably contain: a Wharton study he cites suggests reviewers followed AI advice nearly 80% of the time even when it was instructed to lie confidently. His fix is zero-trust, multilayer verification that checks generated code by methods independent of the model that wrote it.

Issues that covered it