Yadnesh Salvi

Story · arXiv

SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs? (arXiv)

paper · Story page

Small square tiles pour from a hopper through a coarse sieve without catching, then land on a much finer sieve below, where about a third are held on the surface and the rest fall into a tray.

Benchproofer turns 500 real SWE-bench Verified issues into formally verified tasks, admitting an instance only once mechanical and adversarial gates agree on its specification. Across two frontier models, a quarter to a half of test-passing patches admit counterexamples.

In plain words

  • Researchers built stricter checks for software fixes written by artificial intelligence.
  • Their system writes precise rules for each repair, then uses mathematics to check whether the repaired program follows them.
  • Across the systems studied, a quarter to a half of fixes that passed ordinary tests broke those rules.
  • For software developers, these checks can catch errors missed by the examples covered in ordinary tests.

Appeared in

Subscribe

Get the brief in your inbox

Pick daily, weekly, or both. Nothing is gated either way: every issue is on the site and in the feeds.

  • Weekdays at 8:45am IST, one lead story and 6 to 9 items.
  • Sundays, an argued synthesis rather than a recap.
  • One click to leave, and quiet days say so in the subject line.
How often

Weekdays 8:45am IST + Sundays. Unsubscribe in one click.

You're asking for The Agentic Brief by email at the cadence you picked. You can unsubscribe in one click from any issue, and your address is never sold or shared.

Subscribe by RSS →