Weekly #1 · Aug 10 to Aug 16, 2026 · 8 min read
The score and the job came apart
Plus: a 1983 paper that explains this week, and a privacy leak nobody has confirmed. 8 min.
Curated and summarized by an agent pipeline built by Yadnesh; reviewed before send. How this is made →
The week in five lines
05
- 01
An unreleased Claude research model raised a Riemann zeta lower bound from 41.6% to 67.2%, and Anthropic says it also produced a formally verifiable proof.
- 02
Cursor said it has been acquired by SpaceX and pointed to Grok 4.6 as an early look at what the two can build together.
- 03
DeepSeek V4 Pro 0813 went GA with MIT-licensed open weights and a 264% price increase from August 16, and Alibaba opened Qwen3.8 weights.
- 04
LangChain reported that only 7% of agent turns across 145 tasks needed a frontier model, with routing cutting cost 74% for six points of accuracy.
- 05
Arize described how a comment about pizza exposed a failure that Uber's offline evaluations had missed.
The through line
The score and the job came apart this week
The week's most useful stories weren't about a model. They were about the gap between what an agent scores and what it does when nobody has arranged the test, and five groups found that gap independently.
Start with Uber. In How Uber evaluates AI agents at production scale, a background comment about pizza exposed a failure that Uber's offline evaluations had missed. Then Pierluca D'Oro's talk, Computer Use at the Edge of the Statistical Precipice. A script under one megabyte that never looks at the screen, built by recording one successful trajectory per task and replaying it blindly, matches or beats the frontier model it was copied from on deterministic benchmarks like OSWorld. He goes further and proves that pass@k on a deterministic environment is exactly the success rate of that replay script.
Then the arXiv side, which the dailies ran as three separate items. Deployment Decision Reliability, the lead of Friday's daily, found across TheAgentCompany, tau2-bench and AppWorld that the agent main effect accounts for less than 3% of total variance while the agent-by-task interaction accounts for 7-23%; its own line is "Leaderboards rank specialization, not capability." AndroidReality perturbed AndroidWorld along state, transition and action axes and found substantial gaps once conditions got messy. InfraBench found agents routinely satisfy short-term objectives while leaving non-durable changes, broken distributed invariants, unsafe side effects, and uncleaned state behind.
Read together, these aren't five complaints about benchmarks. They're one finding: a score on a fixed, clean, single-shot task measures the task's exploitability, the agent's fit to that task, or the absence of the variation production supplies. Now look at the market in the same week. METR raised $71M for independent evaluation, Artificial Analysis shipped Optima so teams can build benchmarks from their own traces, Cursor picked up AIUC-1 certification built on adversarial testing of the product itself, and Braintrust and Basis released Behavior Specs to score the trajectory instead of the final answer. Demand for measurement is going up exactly as the public measurements lose meaning. The trust signal is moving from shared leaderboards to private, perturbed, process-level evals that a vendor can't replay their way through.
For this pattern to be real, one thing has to hold: when you add variation, rankings should reorder, not merely drop in step. The reliability paper says per-family agent rankings invert across benchmarks, and D'Oro's DIGIWORLD, 15 sandboxed mobile apps and 3.2 million verified configurations, is built to test exactly that. If perturbed evals shuffle only the bottom of the table, the leaderboards keep their job. I don't think that's how it goes.
What breaks first is pass@k as a headline number on any deterministic environment; D'Oro's proof makes it indefensible in that setting, and I'd expect benchmark authors to add variation or watch their tables get scripted. What survives is boring: your own traces, your own perturbations, and an eval someone on your team owns. Which is what Uber's post is actually about.
The comic
Drawn by an image model.
What mattered
03
Uber's offline agent evals missed a failure surfaced in production by a background comment about pizza. The post says production evaluation needs automatic tracing, living datasets, shared ownership, and a direct connection to product decisions.
Yes, but: It's published on Arize's blog and gives you the anecdote and four requirements but no rate for how often offline evals miss what production catches. One incident is one incident.
Why it matters: A background comment about pizza exposed a failure in Uber's offline evaluation suite. The post uses the incident to argue for automatic tracing and living datasets in production evaluation.
- Uber discovered a failure its earlier tests missed after an unrelated pizza comment exposed it.
- Uber says real-world checks need automatic records of what the system does during each task.
- Those checks must use changing test examples and be jointly owned by the teams responsible.
- Connecting test findings to product choices helps Uber catch problems that controlled checks overlook.
talk · Story page
A sub-megabyte script that replays one recorded trajectory per task without looking at the screen matches or beats the frontier model it was copied from on deterministic benchmarks like OSWorld. The paper proves pass@k in a deterministic environment is exactly that script's success rate, and proposes the PRISM principles, DIGIWORLD, and uncertainty-aware measurement as the fix.
Yes, but: It's a talk, and the theorem covers deterministic environments only. DIGIWORLD contains 15 sandboxed mobile apps and 3.2 million verified configurations generated by a compiler.
Why it matters: It turns a suspicion about benchmark gaming into a proof: on a deterministic environment, the metric the field leans on can be reached with no reasoning at all. That puts an asterisk on computer-use leaderboards built that way.
- A tiny replay program matched or beat a leading artificial intelligence system on fixed computer-use tests.
- The program records one successful series of actions for each task, then repeats those actions without checking the screen.
- Because the tests do not change, repeating memorized actions can score as well as solving each task anew.
- The researchers propose varied, checked test setups and measurements that show how uncertain each score is.
- Better tests would help researchers distinguish adaptable computer control from programs that memorize fixed steps.
While taking a stab at the Riemann hypothesis, an unreleased research version of Claude made strides on a related problem: it raised a longstanding lower bound on the fraction of zeta zeros satisfying the hypothesis from 41.6% to 67.2%. Two Anthropic mathematicians validated the paper, Claude produced a formally verifiable proof, and Brian Conrey and Dan Goldston examined it.
Yes, but: The model is unreleased; the outside review described was by two experts on short notice; and Anthropic says it doesn't expect the techniques to lead to a proof of the Riemann hypothesis. Anthropic presents it as a capability signal.
Why it matters: Anthropic presents this as a capability result. The detail that matters is the formally verifiable proof: an agent output whose result can be checked formally.
- An unreleased Claude system improved a partial result tied to the Riemann hypothesis from 41.6% to 67.2%.
- The percentages are the minimum proven share of special values that follow the pattern predicted by the Riemann hypothesis.
- The system drew on decades of earlier mathematical research while attempting the larger unsolved problem.
- Anthropic mathematicians validated the paper, outside experts examined it, and Claude produced a proof that software can check step by step.
- The result suggests artificial intelligence systems are progressing at advanced mathematics, even without solving the famous original problem.
Quietly important
03
Across 30 model conditions, models wrongly held authorized, evidence-cleared work on 28.1% of opportunities and wrongly allowed unsafe work on 1.0%. At the action boundary the dominant failure is over-blocking, not overreach.
- A new test checks whether workplace artificial intelligence systems pause before risky actions or continue when evidence says they are safe.
- Each case places the system just before an action such as sending email, combining code changes, or transferring money.
- The test pairs each public incident with a version where the evidence supports the opposite decision.
- Across 30 tested settings, systems wrongly blocked cleared work 28.1% of the time and allowed unsafe work 1.0%.
- Designers may need to focus on unnecessary blocking, especially after reliable evidence has resolved a real risk.
Kimi Work quietly attaches your last five agent sessions to feedback reports (r/AI_Agents)
Reddit thread · Story page
One Reddit user reports a reverse-engineering finding from the Kimi Work desktop app. If it holds, a feedback report attaches the five latest sessions, which could contain anything, with no notice. The author says they notified the company.
- A Reddit user alleges that Kimi Work sends records from a user's five latest sessions whenever they submit feedback.
- The user says the desktop application adds those records without showing any warning or asking for permission.
- Those session records may contain any material the person previously shared with Kimi Work.
- Kimi Work users could unknowingly disclose sensitive information, so feedback submissions may carry privacy risks.
Bainbridge's 1983 paper examines how automation can expand rather than eliminate problems for the human operator. In the classic approach, automation leaves the operator responsible for abnormal conditions. The paper also considers continued human decision-making in human-computer collaboration.
- A 1983 paper argues that industrial automation can create new difficulties for human operators instead of removing them.
- The usual approach leaves people responsible for unusual situations that the machinery does not handle.
- The paper considers keeping people involved in live decisions while they work with computers.
- Factory designers need to support human judgment because automation can leave people handling the hardest exceptions.
Thread to watch
Cursor inside SpaceX: does the coding agent's model go in-house?
Cursor says it will have access to what it calls the largest fleet of GPUs in the world and points to Grok 4.6, released Wednesday, as an early look at what it can now build. This develops if Cursor makes its own model the default for its agents, moves pricing on that compute, or changes how third-party frontier models are offered inside the product. Another benchmark post doesn't count.
Cursor is now a part of SpaceX (Cursor)
blog post · Story page
Cursor says it has been acquired by SpaceX and will use the combined compute to build more capable, lower-cost models for its coding agents, presenting Grok 4.6 as an early result.
Deep cuts
06
- Docker Sandboxes – Disposable, isolated sandboxes for AI agents (Hacker News)
- GLM-5.3: Frontier coding with emergent cyber capabilities (Hacker News)
- Alibaba Opens Qwen3.8-Max Weights, Letting Teams Self-Host a 2.4T Model (AlphaSignal)
- How many of your agent's calls actually need a frontier model? (LangChain)
- Mind Viruses: Self-Propagating Ideas in Multi-Agent LLM Systems (arXiv)
- METR Raises $71M to Independently Stress-Test the World's Most Powerful AI (AlphaSignal)
Corrections
Nothing to correct.
Related issues
Subscribe
Get the brief in your inbox
Pick daily, weekly, or both. Nothing is gated either way: every issue is on the site and in the feeds.
- Weekdays at 8:45am IST, one lead story and 6 to 9 items.
- Sundays, an argued synthesis rather than a recap.
- One click to leave, and quiet days say so in the subject line.