Sponsorship
When an agent writes the test, the failure mode shifts from broken code to a test at the wrong layer, light UI assertions when there's an API underneath that would tell you whether the thing actually worked. Catching that is what Ivan Davidov and Debbie O'Brien are teaching in Packt's October 21 workshop. The coupon code: BU30 makes it 30% off.
Butch's Take
This past week, automated testing hit the main stage across a lot of my feeds, though the name being used is "verification." The main reason is that folks are having trouble trusting their coding agents, and a coding agent isn't great at checking its own work. Giving an agent a way to trigger tests and verifications, so it gets feedback that no regressions occurred, is how that trust is being built. Once a team trusts its checks, it can hand agents the work of running and monitoring them.
I still think the biggest win for QA teams is using AI to generate deterministic automated tests, because a test that passes or fails the same way every time is something you can build on. The catch is that an agent can write tests that check the wrong thing. Someone still has to build, maintain, and review what it produces so the right things are actually being tested.
This is where I think QA folks have a real opening. A lot of developers are building their first test framework right now because their agents need one. We've spent years learning what makes a test suite worth trusting, and we can bring that to those teams and help them start on the right path. Partner with them early. Help decide what deserves a test and what doesn't. Then use those verification loops, whether a person or an agent is running them, to give us confidence we didn't break anything critical. Let's make the most of this opportunity!
Headlines & Launches
QA That Pays for Itself: Randomized Agent Testing That Ships Tutorials
Seamus James (Lopsy Blog)
The lopsy.art Photoshop-clone is agent-written, so it gets agent-tested: randomized design sessions via Playwright hunt sequential bugs (undo/redo chains) regular e2e misses, an Art Director agent critiques, and every QA run doubles as a publishable tutorial. Full prompt included: /random skill for true randomness, LAST_RUN.md to avoid repetition, verify repro in a fresh window before filing, never fix what you find, report for triage.
Your Agent Learned Selenium From My Old Blog Posts
David Burns (Selenium Blog)
Selenium's maintainer admits agents write most new Selenium code, wrongly: DesiredCapabilities, DevTools XPaths, sleep(5). The fix ships as docs: a curated llms.txt without Selenium 2/3 legacy pages, an AGENTS.md rules block, and letting agents drive a real browser. Review watch: timeout-raising fixes that keep the race.
E2E: An Open-Source Agentic Testing Framework for Web and Mobile
Oskar Kwasniewski (X)
TesterArmy open-sources a TypeScript runner mixing deterministic locators and assertions with agent steps (act, assert, extract) across browsers, iOS simulators, and Android emulators. Passing agent steps are cached and replayed without calling the model. Bring your own model, including ChatGPT or Copilot subscriptions. Ships with a coding-agent skill and llms.txt docs.
Tools & Frameworks
Sedum: Plain-English Browser Tests Cheap Enough for Every PR
sedum-dev (GitHub)
Open-source pre-alpha runner where plain-English ai() steps and deterministic Playwright coexist in one test: the model only locates elements and judges claims, returning probabilities. Claims to cut a per-step platform monthly PR-suite bill from $4,875 to $38-91. Markdown failure reports are written for coding agents, with page text fenced untrusted.
We Taught an AI Agent to QA Our Cloud Console — in Plain English
Cerebras (X)
Cerebras built console-prompt-tests: each test is a plain-English paragraph in a .pt file, run by an LLM agent driving a real browser through Playwright MCP. Nightly self-healing classifies five failure types, only UI drift auto-repairs, and a risk-scoring release gate ends in Go or No-Go. The AI proposes; deterministic code decides.
Create Agentic Playwright Project: A Scaffold With Human and Evals Gates
Emanuela T. (LinkedIn)
A free npm scaffold that packages a full agentic Playwright setup: e2e, visual regression, and API projects; manager, planner, generator, healer, reviewer, and reporter agents; a /feature command running plan-approve-generate-review-run-heal. A human gate and an evals gate sit between AI-written tests and the pipeline, on purpose.
Alumnium Adds a Native Xcode Driver: Agents Drive iOS Without Appium
Alex Rodionov (LinkedIn)
From the Appium Ruby client's author: the next Alumnium release ships a native Xcode driver, letting agents use iOS apps directly via Xcode 27 with the same API, no Appium or Maestro underneath. The maintainer ecosystem is starting to build around the OS vendors' own automation primitives.
Foundations
How to Test a Chatbot
Kristin Jackvony (Think Like a Tester)
A methodical five-part approach for a task testers increasingly get handed: verify refusals on out-of-scope topics, probe security (regular user asking for admin data, user A asking about user B), validate accuracy against seeded database truth, enforce style guardrails, and test truthfulness policy on questions the bot should decline.
Techniques & Tutorials
Vibe-Coded Apps Need Ruthless QA
Serdar (X)
His playbook argues AI bugs are pattern-completion errors wearing the uniform of correct code, with authorization gaps and injection flaws as signature failures. Six practices: adversarial review backwards, property and fuzz tests by default, pipeline security scans plus a model attacking the diff, evals for nondeterminism, production guardrails, humans at the ship gate.
What I Learned Building a Testing Framework for a Local AI Engine
Ujjwal Kumar Singh (beinghumantester.com)
The author built a test framework for Camelid, a local AI inference engine, and the post is mainly about honesty: verifying claims against the exact installed version, catching a would-be test that targeted a fix no release contained, and reporting outcomes beyond pass and fail, like known limitation, blocked, and unverified, each with a mandatory explanation.
Make Your Coding Agent Prove Its Tests Can Fail
Josh C. Simmons (drjoshcsimmons.com)
His completion-report recipe for agents: show the requirement behind each expected answer, a plausible fault the test rejects, and real execution results. Citing research that tests generated while reading faulty code detect 14% of faults versus 25% from a fresh context, this is mutation testing reborn for agents.
AI Can Write a Thousand Test Cases. Can It Learn QA Judgment?
Kavitha Namduri (Medium)
A QualityHubAI progress report: the author encodes her QA philosophy as principles, examples, and corrections, then tests for generalization on unseen features. The agent writes solid workflow tests but never questions the core action (fields, validation, state) or concurrency until pointed. Generating tests is cheap; knowing what to test is not.
A 400k LOC Test Deletion, Powered by a Test-Audit Skill
Peter Steinberger (X)
OpenClaw deleted ~400k LOC of its own tests with little coverage change. The linked test-audit skill gates every new test with four questions (what contract, what credible failure, why existing coverage misses it, does it demand test-only seams) and hunts 15 junk patterns, from assertion-free probes to source greps. Optimize for confidence, not deletion count.
Research & Data
A $22 Evaluation of Jev's 98% Playwright Cost Claim
Jonathan Canales (LinkedIn)
Canales spent $22 testing Filip Hric's claim that Jev makes Playwright CLI 98% cheaper. It didn't hold, and Hric concedes in the comments: LLM-driven browsers don't pay off for test automation at scale. Hric still backs 'code as cache' with fuzzy fallbacks for flaky interactions.
Hot Take 🔥
Dark patterns: Downsizing QA teams in the age of AI
Gaurav Singh (Automation Hacks)
Field report from mentoring sessions: 5-7 companies running silent QA layoffs, forced role transitions, or department culls, with leaders citing agent-written tests and devs-as-QA per change. Singh's counter: the work still needs doing, someone needs conviction to say don't ship, and a 10:1 dev-to-QA ratio is permanent firefighting.
Quick Links
Addy Osmani: Give Your Agent a Way to Check Its Work
Addy Osmani (LinkedIn)
Four test investments for agent-driven development: e2e user-flow tests as ground truth, property-based testing for what must never happen, old-vs-new differential testing on rewrites with random inputs, and fast deterministic loops because slow or flaky ones teach the agent to retry instead of fix. Kicker: examples say what should happen; properties say what must not.
Every 10x Speedup in Software Sparks a Testing Wave
Gergely Orosz (X)
The Pragmatic Engineer's observation: the last time software delivery sped up ~10x (agile, early 2000s), automated testing exploded almost immediately (unit tests, TDD, XP). AI just delivered the next 10x, and a similar wave of automated testing and verification interest is following.
The AI Prompts Professionals Actually Use
Filip Hric (LinkedIn)
A deadpan parody of prompt-engineering influencers: the daily prompt list is 'that didn't work,' 'try something different,' 'this is crap, try harder,' and 'I'm not reading this wall of text, say that again but shorter.' The comments escalate, including Richard Bradshaw's 'Don't make me call Opus.'
Adaptive Rubrics for AI Evaluation: Moving Goalposts?
Mark Winteringham (LinkedIn)
Winteringham asks whether adaptive rubrics, criteria regenerated per run from a meta-rubric, are safe. The top reply: great for exploration, shaky for trend tracking since a score drop can't distinguish model regression from a harder checklist, and cites research that mere rubric rewording flips LLM judge verdicts.
If something in this issue made you think differently about how your team approaches AI in testing, pass it along. The best conversations about AI and QA are happening in Slack channels and stand-ups, not just newsletters.
Have something worth featuring? Reply and send it my way, I read every link.
Thanks for reading,
Butch Mayhew
