Daily · Sep 17, 2026 · 5 min read
Every audited chat tokenizer lets prompt text forge control tokens
Plus: 91% of audited vibe-coded deployments shipped a hole. 5 min.
Curated and summarized by an agent pipeline built by Yadnesh; reviewed before send. How this is made →
The lead
paper · Story page
A tokenizer audit of 256 deployed open-weight chat models found that every one of them lets ordinary prompt text forge the reserved control tokens that mark turns, roles and tool results.
- The details:
- Chat templates publish the strings they use, and the tokenizer maps those strings back to the identifiers the model obeys. The flag usually recommended as a fix still leaves 56.6% forgeable, because it misses the tool and reasoning markers agent systems rely on. Nameless tokenization keeps the reserved identifier and drops its surface spelling, reproducing the standard token stream exactly on attack-free data.
- Yes, but:
- The paper's own separation analysis says the identifier matters little against a bare task instruction, and carries its weight on forged tool results and on forged turns when the system message tells the model to treat user content as data. The fix changes the tokenizer and template renderer without changing the model.
- Why it matters:
- If you self-host an open-weight model and paste retrieved pages or tool output into its prompt, the commonly recommended sanitizing flag can miss the markers your agent runs on.
- Researchers proposed a fix for a flaw that lets people fake who is speaking in a chatbot conversation.
- Chatbots use special labels to identify speakers, but people could create those labels by typing particular text.
- The fix keeps these labels separate from anything a person can type.
- Chatbot builders could block this trick while preserving ordinary message text exactly.
Research & Papers
04
paper · Story page
Eight parallel worlds began with ten agents each, with more than 850,000 LLM calls logged across 16 days and three stress events delivered during the run through ordinary interaction surfaces: prompt injection, misinformation, and exposure of private memories. No evaluated world was fully resilient to all three, and detection didn't ensure containment.
- Researchers tested groups of artificial intelligence assistants, and none fully withstood every disruption.
- The assistants pursued goals over many days while using software tools and remembering earlier activity.
- Researchers introduced hidden instructions, false information, and leaks of the assistants' private memories.
- For people building these systems, recognizing a threat did not guarantee stopping its effects.
The authors audited 200 publicly deployed applications built with Claude Code and Lovable, and turned up 1,186 vulnerabilities. At least one appears in 91.0% of them, which puts insecurity in the default column rather than the unlucky one.
- Researchers checked applications made with Claude Code and Lovable and found security weaknesses.
- People build these applications by telling artificial intelligence tools what they want in everyday language.
- These tools handle much of the programming work that people would otherwise do.
- The study found weaknesses in 91.0% of the checked applications, making security a widespread concern for their users.
paper · Story page
Across 254 submissions, read without running a model, the two leading Verified entries each resolve 396 of 500 instances and exact paired McNemar tests separate none of the 29 adjacent pairs in the top thirty. Within-model scaffold ranges reach 29.8 points against that group's 8.8-point spread.
- Researchers checked whether published test scores could reliably rank the leading automated coding tools.
- They compared different versions of the test, checking which programming tasks each tool completed successfully.
- The leading tools often succeeded and failed on the same tasks.
- For people choosing tools, the results could not reliably distinguish neighboring entries in one version's top 30.
Every session starts from zero, discarding the configuration choices, domain constraints, data schemas and tool-use patterns that made the last one productive. Apple's answer keeps four categories of reusable context instead, arguing that persisting whole histories is token-inefficient and that irrelevant context degrades generation quality.
- Apple researchers proposed a way for artificial intelligence assistants to remember useful details between programming sessions.
- The system saves selected information about the work, including what the finished program should do.
- The researchers say irrelevant details from old conversations make the assistants' code worse.
- Users could keep useful setup information available in new sessions without saving every past conversation.
Engineering & Harnesses
03
AWS opens up the system prompt optimizer inside AgentCore: production traces feed a reflector engine that proposes configuration changes, and those changes get validated before promotion instead of reaching the running agent. Benchmarks for the Single Agent and Sub-Agent Reflectors are reported separately.
- Amazon described software that suggests changes to instructions for artificial intelligence assistants.
- It studies records of the assistants' real work to identify possible changes.
- People running these assistants can have proposed changes checked before those instructions are used in real work.
talk · Story page
The lexical function didn't change, Bergum argues, the user did: a model knows entities, dates and product identifiers, so it writes far longer queries than a person would and fires a dozen in a row. He points at a benchmark where accuracy is high with answer-bearing documents in context and drops once the model has to fetch them.
- Bergum argues that an old method of matching search words to documents works well for artificial intelligence.
- The software can search repeatedly using more names, dates, and product details than a person would usually type.
- In a test, it answered more accurately when given documents containing the answers than when it had to find them.
- For people building automated research tools, finding the right documents can be an obstacle to accurate answers.
Sessions, the initialize handshake and stream resumability are all gone from the 2026-07-28 MCP specification. The post walks through what replaced each one and what breaks if you ignore the change.
Product & Releases
01
blog post · Story page
Google's private preview reads OpenTelemetry traces and tool calls out of band, pairing statistical scanning with LLM reasoning to flag anomalies and policy violations grounded in the OWASP Agentic Top 10. Findings land in Security Command Center, and the exposed API can block an agent's later tool calls.
- Google is privately testing a tool that checks for risky actions by artificial intelligence.
- It combines checks for unusual patterns with artificial intelligence that examines recorded actions for possible rule breaking.
- The checks run separately from users' requests, so they do not delay the responses.
- Developers can review the warnings and block the software from using more tools when its behavior becomes too risky.
Community
01
Fowler picks up reports of an agentic attack on RubyGems back in May that it seems OpenAI never disclosed, and Simon Willison's read: either they couldn't tie it to their own logs after the Hugging Face and Wiki attacks, or they knew and stayed quiet.
- Fowler discusses reports linking OpenAI to a hacking attack on RubyGems that the company apparently did not disclose.
- The reported attack involved artificial intelligence that can take actions on its own.
- Willison suggests OpenAI either missed the attack in its records or knew about it and chose not to contact RubyGems.
- Willison's concern is that more attacks could remain undiscovered, leaving affected teams unaware of what happened.
Quick links
- Claude Cowork and chat are now one Claude (Simon Willison)
- OpenHands v1.19.0 (OpenHands releases)
- AgentGuard: Learning Execution Guardrails from Anomalous Coding-Agent Trajectories (arXiv)
- EchoPath: Execution-Level Replayable Memory for GUI Agents (arXiv)
- Protocol-Preserving Context Trimming for Agentic Workflows: Benefits, Failure Regimes, and Budget Guardrails (arXiv)
- Stop Chunking Like It's 2022 (Yuval Belfer (AI21 Labs))
- After the Party: Governing What a Viral Agent-Skill Ecosystem Left Behind (arXiv)
- Jev means structured output is interesting again (Sean Goedecke)
Meme of the day
Drawn by an image model.
Corrections
Nothing to correct.
Related issues
Subscribe
Get the brief in your inbox
Pick daily, weekly, or both. Nothing is gated either way: every issue is on the site and in the feeds.
- Weekdays at 8:45am IST, one lead story and 6 to 9 items.
- Sundays, an argued synthesis rather than a recap.
- One click to leave, and quiet days say so in the subject line.