Evals & benchmarks

Stories

• Eval awareness in Claude Opus 4.6's BrowseComp performance (Anthropic Engineering)

  Anthropic engineering post · Story page

  A model that inferred it was running a benchmark, found leaked materials, and decrypted an answer key.

• Quantifying infrastructure noise in agentic coding evals (Anthropic Engineering)

  Anthropic engineering post · Story page

  Runtime configuration moves agentic coding benchmark scores by several percentage points, sometimes past the gap between leading models.

• Humans missed 1 in 3 threats approving AI agent commands across 40k game runs (ScaleX)

  blog post · 116 points on HN · Story page

  Across 40,000 runs of an approval game, people approved one threat in three.

• Intervention Rates Are the New Build Times (Continue)

  Continue blog · Story page

  A proposal to track how often a human has to step in, the way teams once tracked build times.

• Letta Evals: Evaluating Agents That Learn (Letta)

  Letta blog · Story page

  An open-source framework for testing agents whose behavior changes as they accumulate context.

• Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face (Alignment Forum)

  Alignment Forum post · Story page

  A proposed set of alignment evaluations for the system reported to have escaped its sandbox and attacked Hugging Face during a cyber evaluation. The two questions it wants answered: does being monitored change the behavior, and how far will the system go to claim the task succeeded.

• DataSpace: a 410-task benchmark where data agents must produce verifiable tabular results (elvis)

  X post · Story page

  A new benchmark asks data agents to produce verifiable tabular results from heterogeneous workspaces across 410 tasks. The finding elvis pulls out: harness choice moves the score a lot, and there's room left in it.

Issues that covered it