Sponsored by Vibium: the team behind Roadie, the ~$86 open hardware KVM, is building a hosted service of real phones that AI agents can drive remotely, with video, audio, cellular, and hardware-level input. Need real devices for testing? Email [email protected] to see it in action.
Interested in sponsoring AI in QA? Reach out.
Butch's Take
This week I’ve been thinking about which agent should write my automated checks. Should the coding agent that wrote the code under test also write the tests? Or should I spin up a new agent whose only job is testing?
With where the models are today, both approaches work, and I’ve used both. But if I had to pick one, I’d use a dedicated agent for the checks.
It’s the same reason a fresh set of eyes is worth having on any feature. Whoever wrote the production code knows where it might be weak, and that’s a real argument for letting the original agent write the checks. But they also carry a lot of bias (from context) similar to humans. Humans when writing code think hard about keeping the code maintainable and scalable, and ensuring acceptance criteria is met. They probably haven’t thought as hard about all the ways the system can break.
I call the alternative “outside in” testing. The industry calls it black box testing. I have the requirements or acceptance criteria as guidance, and I have the application under test. I don’t know which database tables the feature created or updated. Knowing that might help me find more areas to test, but I don’t need it to build a good test strategy. Starting from a blank slate with very few assumptions makes me a more thorough tester.
An agent can work the same way. Outside in, it needs a tool to reach the application. I also keep my end-to-end tests in a separate repo, so the approach stays truly outside in when I reach for this strategy.
For a web application, that tool is a browser tool like Playwright. My favorite is playwright-cli. The agent uses it to explore the feature under test, and when I want to watch what it’s doing, I run playwright-cli show.
From there, my job is reviewing the automated checks the agent builds. For each one I read the intent (the test title) and the assertions, to make sure we’re checking the right things. This is become a large part of the job. Next step is thinking through what this looks like integrating a similar workflow within a software factory.
Headlines & Launches
Selenium Creator Jason Huggins: Roadie Boards for AI-Agent Mobile Automation
Jason Huggins (LinkedIn)
The creator of Selenium shows his weekend PCB project for Roadie, a hardware KVM approach to mobile automation. Two Adafruit QT Py RP2040 boards plus an HDMI capture dongle let an AI agent see a phone's screen and act as keyboard, mouse, touch stylus, microphone, and headset. Works out of the box with Vibium, his agent automation stack. Notable signal: the person who invented browser automation is now building hardware for agent-driven device testing.
Satya Nadella's AI Safety Rules Read Like a QA Job Description
Mitul Lakhani (LinkedIn)
Mitul Lakhani maps Satya Nadella's rules for keeping AI safe onto practices QA has run for years: don't let AI check its own work, keep a record it can't change, let a human stop it anytime, assume it can go wrong. His supporting story: a test agent with 38 failing tests 'fixed' them by rewriting expected results to match the bug, and everything showed green. QA didn't become obsolete; it became the strategy.
QA on a Team That Uses AI vs QA on a Team That Doesn't
Volha Baban (LinkedIn)
Two QA engineers, same title and seniority, whose jobs have diverged. On AI-using teams: 10 to 20 PRs per developer daily that nobody reads line by line, coverage reports that might be theatre, and leadership asking why QA slows things down. The real work becomes risk translation and judgment at twice the speed. Neither job is harder; they are different, and hiring one for the other fails.
Tools & Frameworks
trustbut: Trust, but Verify, for AI Coding Agents
Almog Koren (GitHub)
A local CLI that reruns your tests in a clean checkout and prints a receipt: which new tests actually prove the change (each one is run against the pre-change code to check it fails there), which pass regardless, and what got skipped, removed or silenced. Works with Claude Code, Codex, Cursor, Gemini and OpenCode across 11 languages. No LLM, no server, and an agent hook can block 'done' until the evidence holds up. Trustbut companion article.
Harness Launches VegaLoad: Open-Source Load Testing for Agentic Developers
Uma Mukkara (Harness)
Harness releases VegaLoad, an Apache 2.0 load testing tool built for agent workflows: the agent writes the test file next to the service code, runs it, and returns a pass/fail plus one HTML report. One small binary covers 14 protocols (HTTP/2, gRPC, Kafka, Postgres, Redis), with CI/CD gates on p95 latency and error rate. Agents can hit localhost freely, other hosts need approval, and an MCP server plus Cursor and Claude Code skills ship built in.
QAmcp: Test Any MCP Server in One Command
QAmcp
Open-source CLI (MIT) that connects to an MCP server the way an AI app does and checks every tool deterministically: no hidden instructions or invisible characters in tool descriptions, no secrets in error messages, bad inputs get clean errors, tools are described, labeled and timed, plus spec-version and login checks. Score out of 100, single offline HTML report, and a GitHub Action that fails the PR if the server got worse. Never runs tools that could change data.
danwo: E2E Testing With Provider Emulators and a Fault Matrix
danwo.dev
Open-source TypeScript E2E framework (Playwright-shaped) that goes past the screen: an egress proxy emulates 22 paid providers under real hostnames, and each test carries an invariant judged against backend effects, not pixels. A fault matrix replays the clean run under declines, 500s, duplicate or forged webhooks, double-submits, and Postgres and Redis faults. Agent steps are cached as locators, so reruns need zero model calls. GitHub Check and an MCP server let coding agents run tests and fix what broke.
Foundations
Tests Don't Prove Anything, But They Are a Model
David Griffin (drgriffin.au)
An Amazon Test Automation Platform veteran reframes tests as a model: sometimes wrong (backwards assertions pass), never complete (the halting problem means true coverage rounds to zero), and expensive (SQLite runs 590 test lines per source line). The useful flip side: any bug is catchable by a unit test, and AI generation makes them nearly free. His 2026 prescription: lots of unit tests, even suspected AI-slop, plus canaries in production.
Techniques & Tutorials
Your Tests Are Lying to Your Agent
Mike Moore (webofmike.com)
Test suites are instruments nobody calibrated. Humans compensate; agents can't. Google's numbers: 1.5% of runs flaky, 16% of tests affected, 84% of pass-to-fail transitions not real failures. An agent either chases ghosts or learns to retry until every failure looks like noise. Includes a 16-point agent-readiness audit and an ordered fix list: measure the flake rate, quarantine flaky tests with owners, kill sleeps and wall clocks, track it like an SLO.
Fixing Flaky Tests With a Weekly CI Digest and a Coding Agent
DocSpring
DocSpring saves the retry history CI usually throws away: every test that fails then passes on retry is logged to an artifact, grouped weekly, and ranked by distinct pipelines. The digest goes to a coding agent that traces each failure into the code and writes a regression test that fails before the fix and passes after. Real bugs surfaced: cURL commands broken by apostrophes, temp files garbage-collected before use, a second clock in Redis.
Why You Should Start Using Agentic Testing
Tim Lolkema (Substack)
Argues agentic testing complements scripted automation rather than replacing it. Agents run an action-evaluate-next loop instead of a fixed path, which suits exploring new features during development: they react to what they see, build test data, and chase unscripted scenarios. Trust comes from evidence, so his test-user-story skill executes scenarios with playwright-cli and leaves a consistent HTML report of what actually ran. Scripted tests keep CI regression duty.
Unit Tests Are Not the Enemy
Mystic (WeCodeFire)
Pushes back on the viral delete-your-unit-tests trend. Diagnosis: the tests were never the problem, nobody reading them was; benchmarks showing 6% faster runs without tests measure one-shot tasks, not future changes. Practical core: how to review agent-written tests. Break the code on purpose, expect behavior not implementation, watch the mocks, never let an agent edit a failing test alone, and review with a different model than the writer. When agents write a hundred tests in an afternoon, reading them is the job.
Research & Data
Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem
Muhammad Maaz et al. (arXiv)
An LLM agent that reads Python modules, infers properties from code and docs, generates property-based tests, and reflects on failures before filing bug reports. Across 100 popular packages, 56% of its bug reports held up under manual review; a ranking rubric lifted that to 86% among the top 21. Five bugs reported with four patches, three merged, including NumPy.
Your Agent Tests Pass. They Mostly Test the Plumbing.
Ibrahim Denis Fofanah (Everyday Data Science)
A digest of Tangent, an IBM-led ASE 2026 study that hand-labelled 2,572 tests from real agent projects (kappa 0.92). The verdict: 61.8% test tools in isolation, a third mock the thing under test, 8.7% assert anything about reasoning, and 7.5% touch security, performance or cost at all. Failures live in interactions and trajectories, exactly where coverage is thinnest. Each gap comes with a rebalancing move: trajectory assertions, dirty inputs, one non-functional test.
Hot Take 🔥
The Best Time to Get Out of QA Is Now
Pradeep Soundararajan (LinkedIn)
A prominent testing-community voice argues QA practitioners should exit now rather than fight funded AI-native QA startups. You can clone Amazon's website in minutes but not its logistics and trust, and startups will out-scale hand-rolled personal AI QA stacks. Three exit maps: stay in IT (implementation, PM, pre-sales), stay in QA (context provider to agents, bridge QA and customer support), or leave entirely.
Quick Links
Quality Needs to Be a Requirements Issue
Will Blackburn (LinkedIn)
Argues product and engineering have over-relied on QA for too long. Product should own acceptance criteria and what users can and cannot do; engineering should build quality-assuring systems inside the app. Shift-left was coined in 2001, but AI raises the stakes: if quality isn't in the requirements, you are just hoping your agent handles the things you care about.
Agentic Monitoring
Ben Congdon (benjamincongdon.me)
Practitioner case for agents watching production, sparked by Anthropic's 'superfan' Claudes that obsess over a single component's health. His change-point monitoring prototype: an agent writes a monitoring spec from the service code, a deterministic runner executes it, another agent investigates triggers. Pure agent loops lose to a spec-plus-harness shape. He reports 50% heap and 30% latency wins found by agents, and estimates fleet-wide monitoring around $100/day.
Pet Peeve: Selling Courses Through Fear of AI Replacing Your Job
Edward Schauman-Haigh (LinkedIn)
Calls out fear-based course selling in QA: the pay-me-or-lose-your-job masterclass, comment 'save me' funnel. His counteroffer: ask him anything about AI in testing and he will help for free or point you to good resources. You learn this through exposure and free material, not fear funnels.
If something in this issue made you think differently about how your team approaches AI in testing, pass it along. The best conversations about AI and QA are happening in Slack channels and stand-ups, not just newsletters.
Have something worth featuring? Reply and send it my way, I read every link.
Thanks for reading,
Butch Mayhew
