AI in QA Review

Butch's Take

As I think about the best way to leverage AI in QA, things land into these 3 buckets.

The first is generating deterministic automation code, this is especially helpful when working with enterprise software that have large user bases. This typically leaves to changes in the application to be more conservative. My approach is using a skill with an exploration step, send Playwright CLI or agent-browser out to walk the app, let it read what's actually on the page, then have it write the test and the checks based on what it found. You get an agent that stares at the application and produces tests reflecting real behavior. That works as long as you already trust the behavior in the app is correct.

This leads to the second bucket: is the site doing what it's supposed to do? That's where you bring in requirements, acceptance criteria, product context, and whatever context already lives in the codebase, then let AI do a first pass of analysis that you review.

Third is root cause analysis. A bug report comes in from support and you dispatch an exploration agent to look past the screen into console logs and network requests, which an agent is able to with Playwright CLI. The other side to root cause analysis is when your automation fails in CI, and instead of a human doing the first triage, an agent takes a shot: app regression, flaky environment, or a test that needs updating. You can wire that into the pipeline now and get the feedback before anyone opens the report.

Is there anything I'm missing, let me know, reply to this email or reach out to me on LinkedIn and let me know.

Headlines & Launches

Playwright & Agentic AI: Building Autonomous Test Frameworks
Rex Jones II (Test Automation University)
Rex Jones II's free Test Automation University course on building agentic Playwright frameworks. Five stations: governance rules with a mandatory 'Created By AI' tag, MCP servers driving browsers via the accessibility tree, a Planner/Generator/Healer tri-agent lifecycle, token-efficient Playwright CLI skills, and an assembly-line architecture with human-in-the-loop gates.

Apparently We Need a Testing Mindset After All
Keith Klain (LinkedIn)
Keith Klain: after years of 'testing mindset is a myth' pushback, AI became the developer and the industry rediscovered evaluator independence. LLM-as-a-Judge research names the old problem: judges favor their own generations (self-preference, 2024) and 'preference leakage' biases them toward related models (2026); it's structural relatedness, not bad prompting. Critical distance became evaluator independence, builder became generator, tester became judge. Independent testing was always someone occupying a different epistemic position. Not a myth.

ClickGap: Autonomous QA for ClickHouse | ClickHouse
Lareb Zafar (ClickHouse)
ClickHouse's autonomous QA agent reviews every merged PR: designs and executes real tests, bisects regressions, files issues and PRs with no human on the send button. Five months in: ~500 issues, 200 coverage PRs, most closed as fixed. Most of the machinery exists to earn trust: adversarial review, ten evidence gates, outcome-based throttling. Making the bot worth listening to is the real product.

How to Set Up GrokBot for QE: A Practical Architecture
Vallalarasu Pandiyan (LinkedIn)
Vallalarasu Pandiyan's five-step setup for running QE on GrokBot's persistent AI teammates (browser, filesystem, terminal, plugins): one focused QE bot, the real toolchain (Playwright, Git, Jira, CI), specialized agents for analysis/design/execution/RCA, rich QE context, and governance before autonomy (read-only first, human approvals, no credentials in chat). Honest gap list: needs QE-specific reasoning, reliable RCA, controlled self-healing.

Tools & Frameworks

Agentic Playwright: Hand Your AI the Test Rulebook
Ivan Davidov (GitHub)
Ivan Davidov open-sourced Agentic Playwright: a Playwright + TypeScript scaffold with the AI rules baked in. A 'Constitution' plus 17 skills get auto-loaded by Claude Code, Cursor, and Copilot, and a write-time hook blocks hard waits, XPath, and loose schemas before files are written. Agents must stop and ask when confidence drops below 5.

Cypress Tap: Closing the Agentic Test Loop
Kevin Korenhof (LinkedIn)
Cypress released Tap, a set of CLI commands that lets an AI agent take control of an open Cypress session: run tests and gather results in machine-readable format. Korenhof tested it with a minimal prompt (starting URL plus a one-line test description). The agent wrote the test, ran it, failed on a missed second page, inspected the DOM, added the missing steps, and reran until green, all without human intervention. Part of the Cypress core app, no Cloud subscription needed. The write-run-analyze loop is now closed inside Cypress.

Cypress Tap: Closing the Loop Between Your Agent and Your Suite
David Ingraham (Medium)
David Ingraham walks a full agent-driven diagnosis using cypress tap: sessions, specs, run, reporter, then pin/dom/aria/inspect to see the page exactly as it was when the test broke. His framing: the loop's step 3 (somebody reads what happened) was always the human; tap makes it text. Honest limits: open-mode only, no CI/headless story, runner must stay open.

qpilot: An AI Agent That Runs Manual Test Cases in a Real Browser
broxhq (GitHub)
Open-source agent (built with Claude and Playwright) that executes plain-text manual test cases in real Chrome. Paste the messy Confluence-style steps, no format required; it opens the browser, runs each step, and streams pass/fail/warn with evidence from the page plus failure screenshots. No test code or selectors to maintain: it reads the page as an accessibility tree each action, so nothing goes stale after a redesign. OTP or captcha pauses and asks you, then continues. Runs locally, works against staging, supports Anthropic or any OpenAI-compatible endpoint.

Foundations

Building Effective Agents
Erik Schluntz, Barry Zhang (Anthropic)
Anthropic's canonical agents primer: workflows are LLM calls orchestrated through predefined code paths; agents direct their own process and tools. Five composable patterns (prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer), plus the discipline most agentic-testing pitches skip: find the simplest solution possible and add complexity only when it demonstrably improves outcomes. Still the vocabulary everything else in this issue assumes.

Techniques & Tutorials

AI Testing SDLC, Workflow #1: Regression Tests Before the PR Merges
Artur Bikulov (LinkedIn)
Artur Bikulov is building an AI testing SDLC workflow by workflow; #1 reads PR diffs, compares changed behavior against the E2E catalog, coverage, and test history, then has a cheaper writer agent fill only the proven gaps. Generated tests start @regression-only, never join smoke or deploy gates until human-reviewed, and the agent never approves its own output. Lane ships advisory with a kill switch, trust earned via useful-vs-noise metrics: gaps caught, false reds, flake rate, reviewer rejection rate.

Playwright AI Defect Management With n8n
Vinav Mevada (LinkedIn)
An n8n workflow for automated Playwright failure analysis: webhook takes execution results, AI classifies each failure (application defect, automation defect, environment, test data, or unknown), checks Jira for duplicates, then creates a bug or comments on the existing one with test details, confidence, and root cause. Safeguards: payload validation, structured JSON validation, 80% confidence threshold, human-review path below it, mock mode. Goal: cut manual failure triage and duplicate Jira tickets.

Playwright AI Testing Lab: Keep AI Out of the Fallback Path
Joan Wang (GitHub)
Joan Wang tested the idea of AI stepping in when deterministic Playwright automation hits its limits, integrating Midscene as a runtime fallback. It changed her mind: AI fallback adds different timeout characteristics, nondeterministic model behavior, external model/network failure modes, separate diagnostics, and can hide a real Playwright regression. Conclusion: keep the tracks separate. Playwright handles deterministic regression with blocking CI; Midscene handles semantic and exploratory testing, non-blocking with human review. The lab repo also shows the full framework evolution from vanilla tests to POM, fixtures, auth, and API testing.

Seven Evals for Judging AI-Written Tests
Sunit Poddar (LinkedIn)
Sunit Poddar argues AI-written tests force the question test automation never had to answer at scale: how do we know a test is any good? A suite can be 4,000 specs, 82% coverage, fully green, and catch nothing. His answer: evals. Seven required ones, including defect detection against deliberately broken builds (the only eval that answers 'does this find bugs'), right altitude, flake rate across 5 reruns, change resilience, convention checks, and triage accuracy scored against labeled past CI failures. Then point competing generation skills at the same broken builds and let detection rate decide which survives.

From Testing to Quality Engineering: A Practical Development Path
Adrian How (Medium)
A practical development path from manual testing to quality engineering: strengthen test design before learning syntax, learn the system one request at a time (trace a user action to its API call and correlation ID), adopt Git and CI early, treat assertions as engineering decisions, and investigate failures rather than just reporting red. AI is framed as bounded assistance: it can draft scenarios and explain code, but the person contributing the test owns why the assertion is sufficient.

Hot Take(s) 🔥

QA Is Becoming the Delivery Bottleneck, CTOs Say
Mike Cunningham (LinkedIn)
Recruiter signal: CTOs report QA and testing are now the delivery bottleneck. AI multiplied code output and testing demand, while dedicated QA mostly disappeared. The comments carry the post: Dmitry Loukine runs agentic quality engineering (agents write and run tests and triage failures, humans keep judgment and critical paths), Pierre Bergamin calls dedicated QA a symptom of org-design failure, Oleg Kobets says the discipline hasn't changed, requirements versus product, and 'nothing beats manual testing.'

Unpopular Opinion: QA Engineers Are Needed More Than Ever
Ismail Asci (LinkedIn)
Ismail Asci's contrarian take: reviewing AI-generated code, with or without humans, is a losing game, and most LGTM reviews were never real anyway. His answer is not more review but different mechanics: stronger manual testing at every step of product development. At nu:legal they cut manual code review to near zero, trading it for quality patterns elsewhere.

Your Value Is Not Your Role: Bradshaw on the Reshuffle
Richard Bradshaw (LinkedIn)
Richard Bradshaw on testing's redistribution: agents free developers to test more, and 'we shouldn't care who does the testing, just that it's done well.' QA-as-process is having its spotlight moment across the whole SDLC. The closer is the part to keep: developers adding your skills 'to take your job? Maybe. But if they are adding your skills, maybe you can add theirs and take their jobs? Or be ready for the new jobs.' Your value is not your role or title.

Think First, Prompt Later: An SDET's Anti-Complacency Tip
Andrejs Doronins (LinkedIn)
An SDET's top tip for using AI: think and research things yourself first. As AI speeds up work, he catches himself slipping into complacency with faster, shorter prompts, until the model's 'you're right, I went down the wrong path' reveals the drift. He still types prompts instead of using voice as a forced throttling mechanism: +5 minutes of thinking saves an hour at full speed in the wrong direction.

AI Plays Tic-Tac-Toe and Misses the Obvious Move
Preeti K. (LinkedIn)
Humor post from the 'AI won't replace testers' genre: an AI plays tic-tac-toe, misses the obvious blocking move, and acts pleased with itself. Tester punchline: 'Congratulations, you've successfully created a bug.' Worth noting the source video is from 2023, GPT-3.5 era, as Butch pointed out in the comments; the same stunt on a current model might land differently.

If something in this issue made you think differently about how your team approaches AI in testing, pass it along. The best conversations about AI and QA are happening in Slack channels and stand-ups, not just newsletters.

Have something worth featuring? Reply and send it my way, I read every link.

Thanks for reading,
Butch Mayhew