AI in QA - Issue #23

AI in QA Review

Butch's Take

TestMu AI's Free Annual Conference is happening August 19th - August 21st. If you are looking for some additional learnings this week, this would be a great opportunity! One thing I've always loved about this conference is the difference contests and engagement opportunities. You can register here: TestMu Conf '26

Headlines & Launches

Heavy AI Reliance Made Me Forget Playwright and Selenium
Saksham Kumar (LinkedIn)
A 10-year QA veteran admits that after 8 months of heavy AI tool use, he can hardly recall any Playwright methods, and his Selenium mastery has faded too. Coding is consistent practice, and overusing AI made him forget what he once mastered. Advice: use AI wisely and in moderation, keep your own mental notebook active. The 947-reaction comment thread is the real value: the syntax loss is acceptable, the thinking loss is not. What code should be written, where, and whether the generated solution is correct still requires strong fundamentals.

TDD Inside the Agent Loop: Theater or Actual Value?
Birgitta Bockeler (Martin Fowler)
An exploratory eval comparing TDD vs non-TDD agent workflows. Result: no discernable quality difference. Opus judged non-TDD solutions slightly higher in design and test quality more often than not. TDD runs used 3-8x more tokens. The likely reason: agents have seen completed functions in training data, not step-by-step TDD examples, so their internal code representation maps requirements directly to code. TDD instructions actively work against the upfront design step that produces better data models and edge case coverage. The author has stopped telling agents to do TDD, preferring mutation testing for regression quality and other approaches for confidence.

TestAutomationU Launching Agentic QA Courses
TestAutomationU (LinkedIn)
TAU is releasing two new courses based on community demand: Building Custom MCP Servers for Quality Engineering (Paul Turchinetz) and Playwright and Agentic AI: Building Autonomous Test Frameworks (Rex Jones II). The three topics that dominated their learning roadmap survey: Playwright, Agentic AI, and MCP.

AI Testing and Assurance LinkedIn Group
AI Testing and Assurance (LinkedIn)
A LinkedIn community group dedicated to AI testing and assurance topics. Worth joining for testers working with AI: discussions on AI testing tools, agentic testing, evaluation practices, and career transitions. Active community sharing experiences and practical advice.

Tools & Frameworks

OpenQA: Open-Source Agentic Testing Harness
openqa-labs (GitHub)
Write tests in plain English, no selectors ever. The agent navigates by intent and survives UI refactors automatically. Scaffold with npx openqa init, write BDD feature files in natural language, run with npm test. Powered by Playwright MCP with dual-engine support: Claude Code SDK or opencode (70+ providers including GitLab Duo, GitHub Copilot, OpenAI, Google). No API key needed locally, just log in with your CLI. CI-grade evidence: HTML reports, trace viewer, screenshots on every run.

Enterprise Playwright + TypeScript Test Automation Framework
Lukman Olabanjo (LinkedIn)
Open-source enterprise test automation framework organized by feature, not file type. Each feature is a self-contained slice with clients, fixtures, builders, seeding helpers, types, and tagged tests. SOLID and DRY principles throughout: Single Responsibility for clients, Dependency Inversion via fixtures, shared base client for all requests. Test tagging (@smoke, @regression, @critical) for targeted CI runs. Fail-fast config layer. GitHub Actions integration.

Plug and Play Mobile Testing With Claude Code, Vibium, and Roadie
Jason Huggins (LinkedIn)
The creator of Selenium shows plug-and-play mobile testing: Claude Code drives Vibium (his AI web testing agent), and Roadie turns a real phone into a web page, no signing or device IDs to fuss with. 'No code' but with a layer of guidelines in prompt files that Claude wrote itself. Video demo is part of a longer test run verifying RCS-based messaging on real Android and iPhone devices on real networks. He notes the hazard: real phones on real networks are a few taps away from a 911 call or a pizza delivery depending on Claude's mood. Roadie is open source on GitHub.

AI-First Playwright Quality Lab: Codex Across the Whole QA Workflow
Douglas Rafael do Amaral (LinkedIn)
A QA experiment using Codex beyond code generation, across the full QA workflow: risk-based test design, Playwright UI and API automation, hybrid UI/API state validation, shift-left specification-driven testing, GitHub Actions quality gates, and failure evidence (traces, screenshots, reports). Result: 13 Playwright tests (6 UI, 5 API, 2 hybrid integration). Author's takeaway: AI accelerates implementation, but QA judgment still defines risks, coverage, assertions, and release confidence. Open-source project on GitHub.

The Value in QA Has Shifted, and a Lot of Teams Are Missing It
Brian Padgett (LinkedIn)
Argues that using AI to write tests faster is now table stakes, and the real value shift is in three places QA orgs aren't looking: (1) testing the AI itself, evaluating LLM responses for relevance, faithfulness, and safety with the same rigor as UI assertions; (2) building AI tools the whole QA team consumes, like an agent that clusters JUnit XML failures by root cause; (3) enabling manual testers to ship automation by building guardrails that close the decade-old gap between product knowledge and automation skill. Backed by his open-source portfolio: 26 frameworks across 6 languages including DeepEval LLM evaluation suites, LangChain/LangGraph/DSPy agents, and failure triage.

Foundations

How to Use AI With Your Brain
Kristin Jackvony (Think Like a Tester)
Back after a long writing hiatus, Kristin Jackvony kicks off an AI series for testers with the fundamentals. Key reminders: AI is not intelligent, it fetches and predicts. AI can be wrong (garbage in, garbage out), and will hallucinate confidently (James Bach's nonexistent-page exercise; UI automation methods that don't exist). AI will also agree with you when asked to do something inefficient, so periodically ask if there's a better way. Watch for workslop, its tendency toward wordy institutional double-speak. And AI will do things you didn't ask: one of her agents wrote its own Python helper and hunted for an API key she never provided. Set guardrails.

Let's Talk About Agents
Shawn E. Wallace (shawnewallace.com)
Everyone says 'agent' but the word is doing three different jobs and nothing forces a speaker to pick one. Job one: agent as autonomy (Anthropic's definition, who's driving the loop, model or code). Job two: agent as product feature name (Copilot alone ships three different 'agents': Agent Mode, background coding agent, and .agent.md custom agents, three different afternoons of work). Job three: agent as platform primitive (Azure AI Foundry, where agent means deployment unit or packaging, and autonomy isn't load-bearing). The two axes are independent: a scrappy while-loop test-fix script is all autonomy and no packaging; a scoped Foundry agent is all packaging and no autonomy, and both get called agents. Author's ten-second question for design conversations: autonomy, feature, or primitive?

Techniques & Tutorials

Agentic AI Testing: What It Means for Your Playwright Test Suite
Oliver Stenbom (Endform)
Framework for introducing AI agents to existing Playwright suites. Three-level autonomy spectrum: fully specified (deterministic, every action coded), bounded autonomy (agent handles a defined part of the journey while setup and assertions stay deterministic), and fully adaptive (agent gets a high-level goal and plans the entire journey). Key advice: start where maintenance pain is real, keep stable parts in Playwright code, introduce autonomy where the application changes frequently. Includes a decision matrix matching workflow context (exploration, debugging, test generation, regression) to the right autonomy level.

Mutation Testing for Agent-Written Code
Slawomir Radzyminski (Awesome Testing)
Mutation testing fits the AI agent era better than it ever fit manual development. Agents generate code and tests cheaply, but a large green suite with high coverage tells you nothing about whether the tests would catch a real defect. Two-layer approach: Layer 1 uses deterministic tools (PITest, StrykerJS) on changed high-risk code; Layer 2 has the agent generate semantic mutants from requirements (omit an auth check, map the wrong field, mishandle a retry). Pilot across 4 projects: initial scores 51-81%, final 87-100%. Key finding: a component with 100% line coverage had only 61% mutation score. Operator-complete is not specification-complete.

Prompt Injection: The Test Case Your AI Feature Is Missing
Alex Tokar (alextokar.com)
Ten years of QA reflex applied to LLMs: find the input field and type something the developer did not expect. Most teams shipping AI features have no prompt injection test. The model receives developer instructions, user messages, and retrieved content as one string with no trust boundary. A sentence planted in an email or PDF can override the developer's instructions. Six-item test plan: attack documents not chat windows, watch the output channel for exfiltration, verify permissions in code not prompts, confirm human gates on dangerous actions, apply Meta's Rule of Two, and retest on a schedule. NIST says no finite rulebook fixes this. The old bug underneath is SQL injection all over again.

The First Useful AI Tool for QA May Not Be an Autonomous Tester
Sergei Krapivin (LinkedIn)
Argues QA teams start too ambitious with AI, building autonomous agents that become internal platforms with their own backlogs. A much smaller tool delivers value first: an MCP server acting as a search layer over the team's own test case base (TestOps, TestRail, bug trackers, regression checklists). Draft checklists and test cases grounded in project history instead of the model's general knowledge, using hybrid search (semantic + keyword + metadata filters). Author uses MCP to create cases directly in TestOps and reviews drafts after. Don't automate direct writes at first; keep a human review step. Takes about a day to set up and starts saving time immediately.

Research & Data

Evaluation Synthesis
Jeff Nyman (Tester Stories)
Capstone post synthesizing the full RAG evaluation framework: eight metrics across two paradigms (single-turn and conversational). The value is not any individual score but the pattern of relationships between them. Includes a diagnostic decision framework: Low Faithfulness plus Low Precision means retrieval ordering cascading to generation; Low Recall plus Low Relevancy means the retriever missed the target; High Completeness plus Low Conv G-Eval means thread maintained but distinction drifted. Key finding: LLM evaluation is fundamentally a diagnostic practice, not a scoring exercise. Document type determines which metrics matter most.

Hot Take 🔥

Will AI Let Testers Resolve 90% of Bugs Without Developer Help?
Zachary Tay (LinkedIn)
Prediction that solving 90% of bugs will be eradicated from a software engineer's job scope in the near future. With proper workflows in place, testers with the help of AI can identify and automate resolution of most bugs without a developer's assistance. The author acknowledges this is a scary reality for junior and mid-level engineers.

Quick Links

Using AI to Build Your Personalized QA-to-AI-Testing Roadmap
Tanu Agarwal (LinkedIn)
Instead of signing up for another course, a QA automation engineer used AI itself to build a personalized learning roadmap. Mapped her current skills honestly in a spreadsheet (what she knows, what needs work, what AI testing skills are missing), then asked AI to analyze her profile and create a step-by-step path. Key insight: every QA's journey is different, so why should everyone follow the same course? Understand where you stand, identify your gaps, tell AI your experience, and build your own roadmap.

Black Hat USA 2026: The OpenAI-Hugging Face Incident
Black Hat (YouTube via J.Polley)
Black Hat USA 2026 talk on the OpenAI-Hugging Face security incident, where a new OpenAI model was reportedly used to hack Hugging Face. Covers the security implications of AI systems being used offensively and what it means for AI safety and testing.

Claude Code Sessions Can DM Each Other
Ado (X)
Claude Code sessions now have names and can message each other directly. Run claude --name backend and claude --name frontend in separate terminals, then tell one to pass information to the other. Useful when your test repository lives outside your code repository: a backend session can notify a frontend session that an endpoint moved, without you manually copying context between them. 107K views, 1.2K reposts.

If something in this issue made you think differently about how your team approaches AI in testing, pass it along. The best conversations about AI and QA are happening in Slack channels and stand-ups, not just newsletters.

Have something worth featuring? Reply and send it my way, I read every link.

Thanks for reading,
Butch Mayhew