Story · Giedrius Šteimantas (Oxylabs)
The Missing Layer in Agentic AI (Giedrius Šteimantas (Oxylabs))
talk · Story page
A 200 doesn't mean the page is real: an agent that forwards ten fetched pages when only three returned content spends most of its tokens reading CAPTCHAs. Šteimantas's rebuild uses a compact-JSON search API for discovery, a scraper that fails loudly when blocked, and a browser only as a last resort.
In plain words
- Giedrius Šteimantas argues that unreliable web access, not artificial intelligence quality, often makes automated assistants fail.
- A successful-looking web response can contain a test meant to block automated visitors instead of the requested page.
- His design uses small search results to find pages, then software that extracts readable text and clearly reports blocked access.
- A full browser is saved for cases where cheaper, faster methods cannot reach the needed content.
- Shopping assistants and similar tools could become less wasteful, expensive, and unreliable when gathering information online.
Appeared in
- Compaction erases agent safety rules, and CLAUDE.md prose isn't a control
Aug 26, 2026 · in the sections
Subscribe
Get the brief in your inbox
Pick daily, weekly, or both. Nothing is gated either way: every issue is on the site and in the feeds.
- Weekdays at 8:45am IST, one lead story and 6 to 9 items.
- Sundays, an argued synthesis rather than a recap.
- One click to leave, and quiet days say so in the subject line.