Story · arXiv
Signal or Noise? A Benchmark Study of Agent Skills in Web Development (arXiv)
paper · Story page
Across 31 public WebDev Skills, 50 projects and 1,000 tasks on four models, injecting the target Skill cut mean Pass@2 by 1.3% to 4.2% and raised token cost by 72% to 394%, with gains in only 17% to 36% of Skill-project pairs. On some models an equally long irrelevant Skill reproduced most of the loss.
In plain words
- A study found that adding web-development instruction packages often reduced task completion while increasing the amount of text processed.
- The researchers tested 31 public packages across 1,000 ordered tasks and 50 projects using four artificial intelligence systems.
- Average success within two attempts fell by 1.3% to 4.2%, while processed text rose 72% to 394%.
- An equally long irrelevant package caused most of the loss for some systems, suggesting extra reading itself became distracting.
- Developers should check whether each instruction package actually helps, because added guidance may waste resources or hurt results.
Appeared in
- Compaction erases agent safety rules, and CLAUDE.md prose isn't a control
Aug 26, 2026 · in the sections
Subscribe
Get the brief in your inbox
Pick daily, weekly, or both. Nothing is gated either way: every issue is on the site and in the feeds.
- Weekdays at 8:45am IST, one lead story and 6 to 9 items.
- Sundays, an argued synthesis rather than a recap.
- One click to leave, and quiet days say so in the subject line.