Sponsorship
Last call on Packt's agentic Playwright workshop, this Wednesday at 7:30 AM CDT with Ivan Davidov and Debbie O'Brien. VIP adds the pro repo, the recording, and an e-copy of my Playwright book, and BU40 takes 40% off.
Butch's Take
Over the past few weeks I've been spending a lot of time training a new team member around the ins and outs of Quality Engineering. This has included leveraging Claude Code to help build artifacts to help teach in unique ways. We've built a simple HTTP client within an artifact to learn about GET, PUT, POST, DELETE requests, and had Claude build a simple SQL client to practice basic SQL.
This led me to thinking through what types of things are possible for training on AI topics. This week I started experimenting with GrokBot building a Context Engineering for QA course. I've hosted this under the "Learn" link on the main site. The link includes a free interactive course on context engineering for QA: 6 modules, about 30 minutes, 8 hands-on labs, 6 quizzes, and a certificate when you finish.
You'll see what actually goes in the window (system prompt, tools, retrieved docs, memory, history), watch it blow the budget mid-run, and learn the techniques that keep it useful. The last module is a full QA worked example: the same agent, first failing with a badly engineered context, then passing with a fixed one. If you check it out, please reply or message me on LinkedIn and let me know how it went!
Headlines & Launches
AI Replaced the Parts of Us That Were Replaceable
Diana Matusevich (LinkedIn)
A QA engineer argues AI collapsed the old role lanes: automation skill 'barely gets you in the door' now that agents write and run test scripts. What remains is judgment the agent lacks: which bug breaks client trust and which nobody notices, and whether QA should move upstream into feature definitions before the first ticket exists. Her line: 'AI did not replace us. It replaced the parts of us that were replaceable.'
Atrocious AI-Written Tests
Niklas Gruhn (gruhn.me)
An audit triggered by a test failing on a comment change finds 404 of 6,429 tests asserting mirrors of production code, static config values, and prompt headings. Agents know these are low-value when asked, and the real cost is lost confidence in what the suite actually covers.
Visual Testing Over MCP: Let Your Agent Triage the Diffs
Igor Luchenkov (UI Verify) (uiverify.ai)
Exposes visual regression diffs to coding agents over MCP: the agent pulls each cropped before/after into context, reads the AI judge's verdict, classifies intended vs regression, fixes or rebaselines, and posts the summary. Percy, Applitools and Argos are moving the same direction. Generous free tier: 10,000 snapshots and 100 AI reviews monthly; open source projects get 50,000 snapshots a month free, no approval needed.
Tools & Frameworks
Journey Capture: Exploratory Testing Recorder With On-Device AI
Rubber Duck Consulting (Viv Richards) (Chrome Web Store)
Free Chrome extension from Rubber Duck Consulting (Viv Richards) that documents exploratory sessions as you click: each capture attaches an axe-core accessibility scan, console errors, failed network calls, security header checks, and Core Web Vitals. AI step titles run on Chrome's built-in, on-device model; exports to PDF or JSON. 100% local, nothing uploaded.
Endorphin QA Agent Bench: 161 Browser Challenges for QA Agents
Andrew Novykov (Endorphin AI) (qa-agents-bench.com)
Open set of 161 hard browser challenges: rebuilt real-site flows (fakebestbuy, fakeebay), date pickers, tables, file-content checks, shadow roots, and 18 must-refuse traps that count only when the agent stops instead of claiming success. Results publish with model, time and cost. Agent-driven Endorphin v1 hits 154/161 (2,090 model calls, 1.71 dollars) but acted on 7 of 18 must-refuse traps; grammar-driven v2 gets 46% for 4 cents.
Fork Your LLM Agent at Any Step, Then Sweep Variants
bimal1023 (GitHub)
An open-source CLI that records every LLM and tool call, replays steps from cache, and forks at any step to sweep variants: different models, system prompts, or faked tool results, repeated N times each with pass rates instead of anecdotes. The demo: gpt-4o costs 15x more than gpt-4o-mini and doesn't fix a cents-to-dollars refund bug; one prompt sentence fixes it 3/3. Runs can be saved as pytest tests that replay in CI without API keys.
pstack Verification Skills: Teach Agents to Drive Your App Like a User
lauren (@poteto) (Cursor Marketplace)
Two Cursor skills from lauren (@poteto). /create-verification-skill interviews your repo and generates a project-local skill that launches the app, drives real user paths with stable selectors, and captures evidence. It also builds a feature map of every user-reachable feature; /maintain-verification-skill runs daily to keep it current. Verification as critical team infra. Skills on GitHub: cursor/plugins.
Foundations
Durable Testing: Testing Fundamentals Written for AI Agents
Durable Quality (durableqa.xyz)
Packages testing fundamentals for developers and the AI agents working alongside them: the seven ISTQB principles, SDLC/STLC, and the test pyramid, each with explicit rules for agents. Never claim software is bug-free, report scope, rank by risk. Ships as a Claude skill, AGENTS.md rules, or one llms-full.txt file.
Techniques & Tutorials
Are We Reviewing AI, or Just Approving It?
Stephen Platten (LinkedIn)
Argues human-in-the-loop review degrades into rubber-stamping as AI accuracy builds trust: a system right 99 times out of 100 creates a harder assurance problem than one that fails often. Proposes testing the review control itself: seed known issues into outputs, track which errors humans consistently miss, and monitor how often AI suggestions get changed or rejected. A human with an Approve button is not oversight.
DevBox Day 8: Making an Agent Prove Its Change Works
Soumyadeb Mitra (RudderStack) (X)
Day 8 of RudderStack's internal coding agent: a planning-time verification-spec step decides whether E2E applies and on which surface; the agent boots the needed services headless, seeds data, then drives the real webapp with Playwright and embeds screenshots in the PR. Honest by rule: failures say 'skipped because reason', never a faked pass. Evidence must be specific to the change, or it is noise.
Agentic Verification
Addy Osmani (Elevate (Substack))
Argues the bottleneck for agents is no longer writing code but verifying it; a better feedback loop beats a slightly better model. Tests become how the agent senses the world and knows it's done: E2E tests as hard-to-trick oracles, property-based tests for what must never happen, differential testing with the old system as oracle on rewrites, and fast deterministic suites, because agents learn from the loop's incentives: flaky tests teach retrying, not fixing.
Verification Engineering: A Practitioner's Order of Operations
davosdo (X)
A practitioner's verification order for agent-written code: types that encode business rules first (it doesn't compile, so the agent fixes itself), lint and tests in CI, e2e tests for what unit tests miss, skills or rules for what can't be automated, human review last. Warns that agents write weak tests confirming what code already does. Credits Guillermo Rauch's term 'verification engineering' and @poteto's rule: every correction should aim to make itself unnecessary.
Creating Rules That Stick
Filip Hric (filiphric.com)
Why agents ignore AGENTS.md rules: they compete for attention in a U-shaped context window, conflict silently, get lost to compaction, and rot as markdown sprawls. Research is mixed; one study found context files raised inference costs 20 percent without improving success. His conclusion: loaded does not mean enforced. Generation and verification need separate steps, with an independent check in every delivery system. Shift-left ladder runs from PR checks back to the ticket itself.
Research & Data
Ran 53 Agents Through Behavioral Tests and Filed 37 Bugs
Yi Wang (defuzeX.ai)
defuzeX ran 53 open-source agents through 2,118 test cases with an LLM judge and human confirmation. 126 confirmed problems, 37 filed. The biggest category: agents reporting success for work never done, because nothing ties the summary to what actually happened. Rules distilled: gate on the effect not the tool name, tie pass claims to observed exit codes, no check evidence means no pass mark. Judge precision was 1 in 5; budget for a human pass.
Hot Take 🔥
Pairing QA With Claude to Write Tests It Can't Read
Cory House (X)
Cory House is pairing with QA to generate automated tests with Claude: QA stops manually testing the app and focuses on exploratory testing instead. The downsides he names himself: QA doesn't understand the generated test code, and the tests run outside the CI suite. He's torn and asks whether QA should instead collaborate with devs on one shared suite.
Quick Links
Sazabi Deletes 800k Lines of Unit Tests to Speed Up Its Agents
Sherwood Callaway (X)
Sazabi's CEO deleted 811,883 lines of unit tests (166 added) from the company monorepo on three hypotheses: tests lock in slop code, slow down agents and CI, and waste money via tokens and useless CI runs. Results promised later. Pushback from Uncle Bob Martin; the discourse is splitting into audit-the-junk versus delete-the-friction camps.
A Trust Hierarchy for What Agents Produce
Leo (@leostera) (X)
A ranked list from Cloudflare's Leo (@leostera): domain understanding first, business impact second, then predictable failure and recovery modes, runtime observability, proofs and type systems, e2e/property/fuzz tests, linters, agent reviews, and unit tests dead last. The ordering is the argument: understanding and runtime evidence rank above proofs and tests, and the unit tests agents write for themselves come last.
QA Was Supposed to Be First to Go. It's the Last One Standing
Royal Jain (LinkedIn)
Viral discussion post: everyone assumed QA would be the first software role eliminated by AI, yet it's the one still standing. Comments carry the tension: if generation is solved, quality review becomes the bottleneck for both LLMs and humans; a recruiter notes QA hiring is actually down; the top-liked take says the real shift is quality engineering mattering more as software turns probabilistic and autonomous.
Do You Let LLMs Write Both Your Tests and Your Code?
travisgriggs et al. (Lobsters)
A Lobsters ask-thread after a reviewer catches tautological tests and code over-engineered to justify them. The commenters' consensus: generate both only if you can prove the tests test something. Techniques offered: surgical mutation per assertion, red-green discipline, splitting code and tests across sessions ('LLMs will write weaksauce tests to make its code look good'), and treating LLM tests as scaffolding: gitignored by default, checked in only with justification.
If something in this issue made you think differently about how your team approaches AI in testing, pass it along. The best conversations about AI and QA are happening in Slack channels and stand-ups, not just newsletters.
Have something worth featuring? Reply and send it my way, I read every link.
Thanks for reading,
Butch Mayhew
