Sponsorship
Interested in sponsoring AI in QA? Reach out.
AI in QA Review
Butch's Take
This week I had a chance to watch a video from one of the @Grokbot developers, Lauren Tan(@poteto on X) who described her adoption path on utilizing AI for coding which I found very interesting. She shared the below graphic and shared how her journey from managing a few agents being human-in-the-coding-loop to human out of the loop managing hundreds of agents agents came down to finding ways to trust the output utilizing "Verifications".
As the video continued, Lauren shared 2 skills which are a part of pstack, a skills pack to ship high quality code with AI tools, to help with verification work.
/create-verification-skill(link) Run this skill in your repository and this skill searches through your codebase, and works towards building a new skill that will provide "Verifications" for your site./maintain-verification-skill(link) This skill will help keep the newly created verification skill up to date
I ran the /create-verificaiton-skill against a repo I use on a weekly basis, and and a new skill was created, which included a Feature Map of each of the different areas of the UI, along with some programatic verification steps that are a mix of code snippets and instruction for the agent on how to run the application and how to run the verifications. The feature map is an interesting concept as the context that was built and provided is overall how the system functions along with some business logic.
I share this here because this high level "Verification" is cheap, especially with skills that can build runnable skills with a lots of context built in. Drop the create verification skill in a project repo and explore for yourself!
Headlines & Launches
9 Questions From a Senior AI QA Engineer Interview
Anupam Banerjee (LinkedIn)
Interview questions you may run into for testing AI powered products. Anupam shared 9 themes: strategy for AI products, nondeterministic prompts, hallucinations, RAG, eval metrics, golden datasets, evals in CI/CD, prompt injection and bias, and agent tool-calling. If you are looking for a job my advice is to take this as an oportunity to research, learn and form your own opinions on how you would appreach each of these.
Selenium and WebdriverIO Docs Are Now Machine-Readable for LLMs
David Burns (LinkedIn)
David Burns (Selenium core committer, WebdriverIO maintainer) made both projects' docs machine-readable for LLMs: clean markdown via llms.txt, streamlined routes optimized for context windows. Motivation: LLMs writing Selenium/WebdriverIO scripts often emit outdated syntax or hallucinations because heavy HTML templates make docs hard to ingest. Better docs should mean cleaner generated test code.
Test Rocket: Rethinking the Test Pyramid for the AI Era
Øystein Nilsen (DEV Community)
Proposes the Test Rocket as the successor to the pyramid, honeycomb, and trophy. Key insight: unit tests are now the fastest feedback loop inside the agent's edit-run-revise cycle. Shape: static analysis base, substantial unit/integration body, small E2E nose cone, including test types that never survived human cost-benefit calls. Honest caveat: agents writing tests from their own code assert 'the code does what the code does'; fixes include spec-derived tests and adversarial second agents.
Vibe Coding Didn't Change How We Test Software
Shivani Anand (LinkedIn)
Pushback on 'vibe coding changed testing.' Her example: two contradictory prompts 5 minutes apart; the AI didn't flag the contradiction, just picked a version. Ran fine, broke on a double-click. Her frame: the AI didn't fail QA, it became one more thing QA applies to; expected != actual is just a bug, not an 'AI limitation.' Vibe coding made untested assumptions ship faster.
Tools & Frameworks
passproof: Block Agent Claims of Passing Tests Without Runner Output
AziizBg (GitHub)
Tiny CLI hook that blocks coding agents (Cursor, Claude Code) from claiming tests passed unless real runner output (pytest, jest, vitest, cargo, go) appears in the same turn. It doesn't run tests, it blocks the claim: 'All tests passed. passproof: blocked.' This adds a deterministic check to ensure tests actually ran.
Verefi: From Feature Description to Playwright Tests That Actually Run
Phil Chen (GitHub)
Claude Code plugin turning product specs into human-reviewed test plans and executable Playwright tests. Pipeline: testplan, audit (source selectors, testability grade), discover (live browser verification), implement (page objects; refuses to generate from unverified guesses), execute. A lot of the value for me with these tools is inspecting how things work. This gives me ideas about how I can improve my skills and angents.
Foundations
Thinking Differently About AI and Testing
Nate Custer (TTC Global)
Challenges using AI to do the same testing faster. After a mainframe regression dropped from 3 months to a week, the better question is what wasn't cost-effective before LLMs: deferred non-functional testing, LLMs as code translation, bug-pattern analysis, property-based testing at CI scale, formal verification for web apps. There are also some interesting thoughts around confirmation bias among technical leaders when using AI.
AI-Driven Development Means Test-Driven Development
Jens Oliver Meiert (meiert.com)
Jens Oliver Meiert (ex-Google engineering lead): if you develop with AI, start from tests. Ask for tests first, then the feature, and be explicit, because many models slap tests on after the fact. Tests-first confirms the feature was missing, keeps the AI on a guardrail, proves it was built, and keeps you centered on the problem.
The Death of Test Automation as We Know It
Aleem Shaikh (LinkedIn)
Argues the write-a-test, maintain-it-forever model is outdated: AI that ingests changes, incidents, and risk, then decides what to test, generates and runs tests, and adjusts strategy ('Agentic Quality Engineering'). The valuable tester teaches the AI which 100 scenarios matter, not who automates 10,000. Comments: auditing a decision-maker needs accountability; self-regenerating suites are tough to audit for sign-off.
Techniques & Tutorials
How to Improve Playwright Test Coverage Using Agent Context
Rizel Scarlett (LinkedIn)
Built an app with Codex, then audited the agent-written Playwright suite using captured agent session context (the tool Entire binds prompts, transcripts, and tool calls to Git commits). A fresh agent compared 5 passing tests against the planning checkpoints: they never tested that moving your hand plays a note. The fix fed realistic hand landmarks through the full pipeline.
Using Matt Pocock's /grill-me Skill to Stress-Test a QA Plan, Then Turn It Into Playwright Tests
Filip Hric (LinkedIn)
Filip Hric used Matt Pocock's /grill-me skill (800k+ installs) to stress-test a QA plan, then turned the result into Playwright tests. The skill interrogates designs like a code review for planning: walks decision trees, surfaces dependencies, forces implementation details. A practical pattern: adversarial interview to find holes in your test plan before writing test code.
Browser Automation with Pydantic AI + Playwright
Pamela Fox (blog.pamelafox.org)
Tutorial on building a manual QA agent with Pydantic AI plus the PlaywrightBrowser capability from pydantic-ai-harness: loads a URL, makes a test plan, investigates reproducible bugs, writes a report. The safety rails are the value: allowed_domains gates navigation and fetch/XHR, private IPs blocked by default, auth via Playwright storage_state instead of your real session, OpenTelemetry traces to audit what the agent did.
QA Testing With Claude AI: A Tester's Real Workflow, and a Reality Check
Reddit (r/QualityAssurance) (Reddit)
A slot and crypto games tester's Claude workflow: test plans and cases per product, automated execution, Claude-drafted bug reports from Jira. The open question: does it scale to multiple products? The top comment is the substance: without guardrails and evals, Claude tests lead to garbage testing. LLMs write code that 'works'; QA finds flaws. Devs feed a snippet and get feature plus unit tests; testers carry system, integration, and acceptance context.
Hot Take 🔥
AI Agents Can Act Like Human Testers and Remove the QA Bottleneck
Stefano Rinaldi (LinkedIn)
Argues manual testing of complex releases is the biggest release-process bottleneck, and AI agents are getting close to handling it: not basic scripts, but agents that navigate the product, check scenarios, and adapt when a button moves. Claims teams can remove the QA bottleneck entirely. Short on specifics; a clean statement of the 'agents replace manual QA' position.
Quick Links
My Tests Are Flaky and Claude Just Recommended Re-Running Them
Filip Hric (LinkedIn)
Filip Hric's tests were flaky, and Claude's recommendation was to re-run them. The AI absorbed the industry's worst habit, since 'just hit re-run' is what training data shows people doing. Andrew Knight's comment: 'We are doomed.' A tidy anecdote for why AI advice on flakiness needs QA judgment.
I Don't See Manual Testing Existing in a Few Years
Keith Burnham (LinkedIn)
A former manual tester argues manual testing won't exist in a few years, and hand-written Playwright tests won't fare much better. Autonomous agents already navigate, interpret interfaces, and test hundreds of paths. The value shifts from typing scripts to reviewing coverage, finding gaps, and directing agents at real risk.
The Hottest Software Engineering Job of the Future Might Be Manual QA
Dmitry Stepanov (LinkedIn)
Argues the hottest software job of the future might be manual QA, 'and I'm only half joking.' Code became cheap; when agents build and attack implementations, the valuable artifact is the description of behavior: what must always be true, what must never happen. The human role is saying 'Nope, that's wrong.' Closing: 'We spent 20 years teaching testers to code. We may spend the next 20 teaching programmers to test.'
Anthropic's ELI5 Claude Skill: Big Pictures, Few Words
Thariq (X)
Thariq (Claude Code team at Anthropic) shares the ELI5 skill Anthropic folks use heavily: 'explain like I'm someone who knows nothing about this topic, using a HTML artifact with big pictures and few words.' Visual explainers instead of prose walls. Installable from the community marketplace; they're debating making it official.
If something in this issue made you think differently about how your team approaches AI in testing, pass it along. The best conversations about AI and QA are happening in Slack channels and stand-ups, not just newsletters.
Have something worth featuring? Reply and send it my way, I read every link.
Thanks for reading,
Butch Mayhew