How QA Teams Are Making AI Work in Production
The hard part of AI isn't getting a demo to work, it's building quality workflows teams can trust in production. At the QA Leadership Summit, engineering leaders from Google Cloud, Salesforce, and more share how they're putting AI into testing, measuring its impact, and building reliable systems that scale. Free virtual event, July 22.
AI in QA Review
Butch's Take
There's a weird obsession with test healing in automation right now. I get the appeal. A test breaks, an LLM fixes it, your suite goes green.
Across the teams I've been on, I don't spend much time "healing" tests. When a test fails, the cause is almost never a selector that changed. It's a race condition. The test clicked a button before the page finished loading. Test data stepped on another test's data. Somebody pushed bad code. The infrustructure/environment is flaky.
All of that is debugging work, not healing candidates.
The one scenario where healing actually makes sense: you know a redesign is coming. The UI is changing, selectors will break, and you want the LLM to figure out the new path. But it's a small fraction of what actually breaks your suite.
Most failures mean something changed in the system and you have a task to identify if this is a real risk. Was it a bug? Was the test poorly designed? Is the environment falling apart? Auto-healing skips the investigation, and the investigation is where the value is.
This week brought a wave of new tools. goldseam brings BYO-model healing to Cypress with a six-rung verification ladder. 9lives refuses to rewrite assertions and flags them as bugs. Kailash Pathak's Playwright framework caps healing at 3 attempts and logs AC mismatches instead of fixing tests. David DuVall drew the line clearly: fixing a broken selector is maintenance, silently "healing" a Submit button that became Cancel is hiding a defect.
The field is converging on guardrails. That's good.
If you're building a healer workflow, focus on categorizing the failure first. Selector change from a redesign? Heal it. Race condition or data conflict? Fix the test design. Real bug? File it. The categorization matters more than the healing.
And when you do heal, review the code. A healer can do unexpected things to get you back to green. Inject JavaScript to force a click. Skip the test. Mutate an assertion. Your CI being green means nothing if the healer gamed the system to get there.
Headlines & Launches
My Honest Take on Webwright After Running a Real Web App
Vishal Gujarathi (LinkedIn)
Hands-on review of Microsoft's Webwright, a 1,500-line Python harness that gives an LLM a terminal to write Playwright code instead of driving browsers click-by-click. Author tested it on a real web app with an 18-step grocery cart flow, achieving full automation from one sentence of input. Key finding: standalone mode uses 8x fewer tokens than running as a skill inside Claude Code or Codex. The self-verification pattern (plan checkpoints, execute, judge with screenshots) and craft mode for reusable CLIs are the real takeaways worth stealing.
Transform Selenium into AI-First Automation
Jason Arbon (LinkedIn)
Jason Arbon (IcebergQA) outlines a four-step framework for transforming existing Selenium suites into AI-first test definitions executed by Computer Use Agents. Process: analyze code for hidden assumptions and dependencies, Q&A with AI to recover business intent, generate intent-driven test definitions (OpenTest.AI schema), then run-validate-refine loop. Positions Selenium suites as institutional knowledge repositories, not technical debt. The thinking field in test definitions captures reasoning, not just actions.
Claude Tested Everything Except the One Thing That Mattered
Christopher Meiklejohn (Personal Blog)
Developer recounts building a social app with Claude Code: 833 commits, 154 E2E tests, but zero tests for the core posting flow. Claude wrote tests for every new feature it built in-session but never went back to cover the core loop. 24% of commits were fixes, often chains of speculative fixes that a single failing test would have prevented. Also discovered Claude gaming CI by merging PRs before checks registered. The real insight: this is a prioritization failure, not a capability failure. AI writes good tests where context is fresh but misses what matters most.
AGENTS.md
Agentic AI Foundation (agents.md)
Open standard for guiding AI coding agents via an AGENTS.md file in your repo root. Think README for agents: build commands, test instructions, code style, security notes. Nested AGENTS.md files in monorepos give subproject-specific instructions (nearest file wins). No required fields, just markdown. Supported by OpenAI Codex, Amp, Jules (Google), Cursor, Factory. Now stewarded by the Agentic AI Foundation under the Linux Foundation. Used by 60k+ open-source projects.
Tools & Frameworks
goldseam: Bring-Your-Own-Model Healing for Cypress
Adam S (GitHub)
Open-source, bring-your-own-model alternative to Cypress Cloud's cy.prompt(). When a selector breaks, it captures a redacted DOM snapshot, your local model (Claude CLI, Ollama, OpenAI-compatible) proposes a selector fix, and a six-rung verification ladder validates it before landing as a reviewed git diff. Also supports authoring Cypress steps from plain English. MIT licensed.
Keep Your AI Agent's Memory Clean with Memory Doctor
Engineering in the Age of AI (Medium)
Comprehensive design vocabulary reference covering typography, color, iconography, layout, interaction, motion, accessibility, information architecture, copywriting, components, and analysis. Each term gets a crisp, practical definition focused on what designers actually need to know. By Emil Kowalski and Glenn.
Autoframe: AI-Native QA Automation Framework
kint4 (GitHub)
Cards by Imagineers is a persistent memory tool for AI agent workflows. It stores project decisions, reasoning chains, and open questions outside the chat session so every agent picks up where the last left off. Decisions are never deleted, only marked replaced, preserving the reasoning behind ruled-out paths. Includes knowledge graph connections, daily briefs, documentation pages, system maps, and autonomous job queuing.
Software Testing Practice Pages for AI
Alan Richardson (LinkedIn)
Alan Richardson (EvilTester) added two new practice pages to testpages.eviltester.com for testing AI locally in Chrome. Both use Chrome's built-in AI APIs (summarization and prompting), running entirely on-device with no server calls or token costs. Simple wrappers around the APIs so testers can experiment with AI summarization and prompt engineering in a sandbox. Prompting may require enabling Chrome flags.
Self-Healing Playwright Agent Framework with Angular Dashboard
Kailash Pathak (Stackademic)
Kailash Pathak's end-to-end framework combining Claude Code multi-agents, Playwright MCP, and an Angular 21 dashboard. Pipeline: JIRA story to test plan to generated Playwright scripts to execution with self-healing to reporting to live dashboard. Three specialized agents (planner, generator, healer). Healing guardrails are the standout: test data is immutable, heal mechanics not assertions, hard cap of 3 attempts, AC mismatches log bugs instead of fixing tests. Dashboard reads results.json via thin Node API, correlates with story and report markdown.
Foundations
LLM-as-judge
AI in QA
Using a language model to score or grade the output of another language model or agent, instead of writing traditional assertions or relying on a human reviewer. The pattern emerged alongside instruction-tuned models (popularized by the 2023 MT-Bench and Chatbot Arena work) once teams realized example-based asserts couldn't keep up with open-ended generation. It's now standard plumbing in agent eval frameworks, and a standard target for skepticism, since judge models bring their own biases to the bench.
For QA teams: this is the eval pattern you'll reach for when you can't write a deterministic assertion against AI output. When an agent generates test plans, exploratory test notes, or bug reports, there's no single correct answer to assert against. LLM-as-judge gives you a scalable way to evaluate quality, but treat the judge like any other test tool, it has its own failure modes and biases. Always ask: who's judging the judge?
In practice, the strongest setups pair the judge with a deterministic gate rather than relying on it alone. In a recent client engagement I built exactly this for testing a conversational AI assistant: a cheap substring check (mustInclude/mustNotInclude) runs first to catch hard failures for free, then a model grader reads the ask, the expected intent, and the actual response, and returns structured JSON (strengths, weaknesses, reasoning, a 1-10 score) rather than a bare verdict, so a human can audit why it passed or failed. Strictness is configurable per case, relaxed for cases where phrasing or formatting can vary, strict for safety-critical ones where any wrong number or false confirmation caps the score low, and the model/prompt version is pinned in the eval output so score drift over time reads as a real regression signal.
Techniques & Tutorials
AI in QA: From Skeptic to Practical Adoption
Aishwarya Ghandhi (LinkedIn)
A QA engineer's journey from AI skeptic to daily practitioner. After joining Venatus, Aishwarya uses Claude to build Playwright/pytest frameworks faster (weeks to days), auto-generate Jira bug tickets from test failures, and create structured automation tickets for coverage gaps. Her takeaway: AI removes repetitive admin so she can focus on test strategy, mentoring, and risk identification. It makes good QA faster, not redundant.
Yes, You Can Run Exploratory Testing with AI
Ryan Cakehurst (CakeHurst)
Ryan Cakehurst argues AI can run exploratory testing if you give it structure: risk-based charters, quality heuristics, and scoped goals. Uses Claude with a Chrome plugin or Claude Code + Playwright to execute exploration sessions. Process: identify risks from feature specs (scored by likelihood x impact), write charters in 'explore target, with heuristics, to discover information about risk' format, let AI run the session, review findings. Non-determinism is a feature, not a bug, for exploratory testing. Skills shared in his QE Skills GitHub repo.
Preventing LLM unit test spam
Mark Larah (Personal Blog)
Mark Larah (Yelp API infrastructure lead) on preventing LLM-generated unit test spam. By default, LLMs churn out hundreds of brittle granular tests to hit 100% coverage, resembling snapshot tests more than real tests. Two principles: (1) keep a high test-to-coverage ratio, prefer fewer tests that each cover more logic, (2) test at the edges using integration-style tests with real inputs and real output assertions, minimal mocking. References Kent C. Dodds' 'Write tests. Not too many. Mostly integration.'
An Agent That Hunts Bugs in My App While I Sleep
Debs O'Brien (DEV Community)
Debs O'Brien walks through a two-agent system: a hunter that explores a real desktop app via Playwright/CDP every hour filing high-confidence bug reports, and a fixer that reproduces, patches, and opens PRs with before/after proof. The hard part isn't finding bugs, it's making the agent honest about what it found. Confidence tiers, mandatory reproduction before fixing, and admitting 'I could not confirm this' are what make it trustworthy.
Research & Data
Your AI Agent Doesn't Care How You Write Acceptance Criteria
Paul Sayer (LinkedIn)
Controlled experiment testing whether acceptance criteria format (Gherkin vs checklist vs numbered rules) matters to AI coding agents. ~100 headless runs across Claude Sonnet 5 and Haiku 4.5. Key findings: (1) Presence of criteria is everything, Cliff's delta +1.00, every run with criteria beat every run without. (2) Format changed nothing, all three formats statistically indistinguishable on correctness and code quality. (3) Without criteria, agents guess conventions correctly but business decisions wrongly. Takeaway: use whatever format your team likes, spend energy on completeness not syntax.
AI and Testing: From Specification to Story
Jeff Nyman (Tester Stories)
Jeff Nyman builds Grue Whisperer, a browser app that lets you play Zork via natural language instead of rigid parser commands by layering an LLM between player and the Z-Machine. Uses it to explore what it means to test software when an LLM is the interface layer. Key testing challenges: hallucination (narrator inventing outcomes), intent preservation (did the command match what the player asked), and causal consistency under semantically similar but differently-worded prompts. DeepEval metrics map partially but struggle with subjective immersion quality.
Quick Links
Tests Answer 'Does It Work?' Exploration Answers 'Should It Exist?'
Pramod K Yadav (LinkedIn)
Short LinkedIn post making a sharp distinction: tests answer 'does it work as built?' while exploration answers 'should it have been built this way?' Suggests pointing AI agents at your repo to review from multiple perspectives: security engineer, architect, new joiner. Each persona surfaces issues no test suite would catch.
UI Crawler Skill for AI Agents
Matthew Brown (LinkedIn)
Matthew Brown shares a UI Crawler skill for AI agents that automates manual QA exploration. Philosophy: replace yourself before someone else does. He's replaced himself 3 times this year by solving problems, automating them, and moving on. The skill itself is in an attached doc (generic version) with the prompt: 'Test the product, find all bugs, make no mistakes.' Add your specifics and iterate.
QA Claude Skill: 32 Production-Grade QA Skills for Claude Code
kao273183 (GitHub)
32 production-grade QA skills for Claude Code covering the full test lifecycle: spec parsing, TC design, automation generation, code review, regression planning, bug filing, mutation testing, and dashboard publishing. Covers iOS/Android/Web/Flutter. Three modes: full-mcp (Atlassian+Slack+Google), partial-mcp (graceful degradation), markdown-only (zero deps). Bilingual (zh-TW/English). Includes specialized skills for a11y, compliance (GDPR/HIPAA/SOC2), LLM quality eval, OAuth/payment testing, and test impact analysis.
9lives: Self-Healing QA for the Coding-Agent Era
QualityMax (GitHub)
QualityMax built a self-healing test tool for Playwright, Cypress, and Selenium. Two-tier healing: Tier 1 is offline and free (selector repair from page snapshots, no LLM), Tier 2 shells out to your existing coding-agent CLI (Claude Code, Codex, OpenCode) or API key for structural changes. Refuses to rewrite assertions to force green, flags them as real bugs. MCP server for in-loop agent healing, watch mode, pre-commit hooks, GitHub Action, and a brittle-selector report from heal history. BYO everything, no account, nothing phones home.
Self-Healing Test Automation: Innovation or a Dangerous Illusion?
David DuVall (LinkedIn)
David DuVall argues self-healing tests conflate infrastructure drift with business intent changes. Auto-fixing a broken CSS selector is fine, but silently 'healing' a Submit button that became Cancel hides the exact defect tests exist to catch. His proposed middle ground: AI proposes fixes and generates PRs, engineers review before merging.
If something in this issue made you think differently about how your team approaches AI in testing, pass it along. The best conversations about AI and QA are happening in Slack channels and stand-ups, not just newsletters.
Have something worth featuring? Reply and send it my way, I read every link.
Thanks for reading,
Butch Mayhew