Sponsorship
Interested in sponsoring AI in QA? Reach out.
AI in QA Review
Butch's Take
This week a peer asked how to get started with AI tools for the workplace. Here is what I recommended.
I've had a lot of great success using Claude Code both at work and on my personal projects. Get started by buying a subscription. The Pro plan for individuals costs $20 USD per month. This should give you plenty of tokens to use in your learnings. Tokens used in this subscription are highly subsidized and really does give you the best bang for your buck.
Next up install Claude Code, the CLI version. This may be a bit overwhelming, but I believe it's the best way to use this tool. Once you have it installed, create a new folder, and open that folder in your terminal and run claude from that directory. You can think of this as your claude "project folder" where you can add specific skills and provide context in CLAUDE.md files. At this point you should be able to ask claude to do things for you... it could be write a script or answer questions, or build a tool, something small worth automating. If you don't have any immediate ideas, it's time to do some learning.
Anthropic has quite a few courses they have released. The 3 I would recommend starting with are:
- Claude Code 101
- Claude Code in Action
- Introduction to Agent Skills
One word of advice, don't just watch these. Open a terminal next to the video and type in the commands along with the video. The act of doing tends to stick more than watching alone.
MCP servers and CLI tools are what turn Claude from a chatbot into something that acts on your behalf: reading files, calling APIs, running a test suite, whatever you connect it to. Most of the time you can just type in a goal with the right model and get good results.
For planning, coding, or working through a problem you haven't solved before, use Opus 5. It costs more, but that's the job it's built for.
Once you've got a skill or workflow that actually works, move it to Sonnet, the mid-tier model, and let that run the repeatable version.
Want a starter prompt to build a simple skill?
I want to create a reusable skill named "execute-testcase", a workflow that will take a test case as an input and execute the test case in a browser using playwright-cli (docs https://github.com/microsoft/playwright-cli). As a part of the workflow if there is missing information such as a url or username/password credentionals ask me for them before proceeding. When exploring the site use the playwright-cli show command so I have some observability on what the agent is doing. The output I want is an HTML test report with the results. Be sure to check the console logs and http requests and report any errors. Also take screenshots and include them in the final report.
copy and paste into claude prompt to create a re-usable skill
Headlines & Launches
Congratulations, QA. We've Been Promoted to Testing the Testers.
Lana Rafaella (LinkedIn)
AI can generate tests, write automation, inspect logs, and review code. But when it confidently reports everything is fine while testing the wrong requirement, someone has to test the tester. The new QA checklist includes hallucinations, bias, prompt injection, robustness, alignment, and evaluating the evaluator itself. A new profession is emerging: AI Evaluation Engineer. Testers have been training for this their entire careers: asking the annoying questions, probing edge cases, and not trusting confident-sounding output.
AI Adoption in Testing Still Lags Behind Development
Sergei Krapivin (LinkedIn)
A paper suggests AI adoption in software testing still lags behind development, which is surprising given how much repetitive work testing has. Most discussions focus on generating test cases, but Sergei finds that the least interesting use. More valuable: reviewing requirements, analyzing failed runs, understanding flaky tests, and deciding what really needs to be retested. If AI helps with one annoying part of the job, that's already a good reason to use it.
Can an SDET Actually Become an AI Engineer?
Andy Johnson (LinkedIn)
The overlap between SDET skills and AI engineering keeps growing: systems thinking, building quality in early, human judgment that no tool replaces. The strongest SDETs who have serious Python capability and are actively building with LLMs, APIs, RAG, agents, and evals are closer to an AI engineering role than they realize. Gaps remain, but the mindset transfers directly. Comments add: the instinct to systematically break things down before trusting output, debugging mindset, and questioning whether a system is actually behaving correctly all carry over.
Tools & Frameworks
AI Is Making Exploratory Testing Scalable
Ronald Veth (LinkedIn)
Use an AI agent with Playwright CLI to explore your application with a clear mission: find console errors, check accessibility, analyze unnecessary network requests, discover hidden features and flows, generate detailed Markdown reports with screenshots, traces, and video. Practical use cases: exploring poorly documented projects, verifying bug fix impact areas, complementing traditional test automation. Comments add depth: Jigar Patel notes playwright-cli saves state to disk between commands so long exploratory sessions don't lose coherence, Andrew Kelly describes risk-driven exploratory experiments with AI.
Agent Plugins 1.0: A Portable Package Format for Agent Skills and MCP
Harald Kirschner (AAIF)
Agent Plugins 1.0 defines an open, vendor-neutral package format so Agent Skills and MCP server configurations can travel together as one portable unit. Instead of rearranging files and rewriting manifests for each client (Cursor, VS Code, Copilot, Codex, Kiro), you package once and any compatible client can discover and load the components. Built collaboratively by Amazon, Cursor, Microsoft, OpenAI, and Vercel. Deliberately narrow in 1.0: standardizes Skills and MCP only, leaves installation, permissions, and trust to each client.
MDCP: MarkDown Context Protocol for Agent-Managed Documentation
Betsalel Williamson (GitHub) Agile Testing Fellowship Newsletter
A new tool on how to store documentation and context in your repo. An Agent Skill that keeps documentation in small, validated Markdown shards instead of monolith READMEs. Agents read one shard at a time, update shards before changing the system, and run checks in CI so docs stay findable and trustworthy.
Microsoft Skill Recorder: Record Once, Replay as an Agent Skill
Microsoft (GitHub)
Desktop app that records your on-screen work session and uses the GitHub Copilot CLI to reconstruct it as an intent plus ordered steps, then builds a reusable Skill or Automation. Record yourself doing a task once (clicks, app switches, URLs, optional narration), hit Analyze, and Copilot turns it into a SKILL.md procedure an agent can repeat and generalize. Prefers native tools (gh CLI, web_fetch) over replaying UI clicks. Outputs to Microsoft Scout, Copilot Cowork, or Copilot Studio. macOS and Windows 11 supported.
Foundations
Where I Still Matter: What Vibe Coding Taught Me About the Future of QA
Aryadevi Neelakantabhattathiri (Medium)
A tester's honest reflection after using AI to build, test, and ship. AI generated test cases faster, ran checks with consistency, and caught code quality patterns. But everything went green and a real user still found a bug on day one. The gap: nobody told AI what correct actually meant for this specific user in this specific context. Five places where human judgment still decides everything: defining what good means, understanding real intent versus stated request, exploratory curiosity (that's weird), empathy for user confusion, and deciding what risks matter. These are judgment calls, not technical tasks that scale.
The Law of AI Confidence
Jason Arbon (Substack)
The AI Confidence Gap is the widening distance between what AI can generate and what organizations can confidently ship. Every doubling of AI artifact generation roughly quadruples the validation work required to ship with high confidence. By 2029, more than half of enterprise AI infrastructure spending will go to validation. Generation becomes a commodity, validation becomes the platform, confidence becomes the competitive advantage. Five testable predictions including: at least one independent AI-validation company will exceed 10B valuation by 2029, and high-stakes AI outputs without machine-verifiable evidence will be treated like unsigned code.
Techniques & Tutorials
My Agentic Framework Had a Hidden Gap. Here's What I Built to Close It.
Srinivas Malempati (LinkedIn)
Built an Agentic Validation Layer (AVL): a deterministic, dependency-free script that re-derives truth from real files on disk to verify agent claims. No LLM judge, no API calls, just file reads and JSON comparisons. Agents must write claims to a manifest before declaring success. AVL checks every claim against actual artifacts: step phrases must exist in the catalog, gate exit codes must match real output, test counts must match real results. First week in production caught a paraphrased domain rule citation that looked correct but wasn't verbatim. Key insight: the policy of requiring AVL before any token caused an agent without execution access to withhold its own approval rather than fake completeness.
Why AI Testing Needs Confidence Scores, Not Just Pass/Fail Results
Rajeshkumar Rajaseakaran Nair (DZone)
AI systems are probabilistic, but most teams still test them with binary pass/fail frameworks built for deterministic software. A 97.4% pass rate tells you nothing about how confident the model was on each pass. Confidence scores unlock four dimensions binary testing cannot: uncertainty mapping (where is the model unsure), threshold calibration (route low-confidence decisions to human review), distribution shift detection (confidence drifts before accuracy degrades), and risk stratification (apply proportional oversight by consequence). Includes practical Python examples for confidence-aware test assertions and production monitoring.
Research & Data
Evaluating Conversations
Jeff Nyman (Tester Stories)
Moves from single-turn RAG evaluation to conversational evaluation. Introduces ConversationCompletenessMetric (did the exchange fully address the user's goal across all turns?) and ConversationalGEval (custom criteria applied across a full conversation). Tests three four-turn conversations against a time travel essay, each targeting a distinct failure mode: consistency across a conceptual chain, semantic drift across sections using the same word with different meanings, and epistemic register (does the model preserve the essay's hedging or sharpen speculative claims into assertions?). Key finding: conversational metrics catch failure modes invisible to single-turn metrics, and both are needed for a complete diagnostic picture.
Hot Take 🔥
Most QA Engineers Are Learning Automation. Few Are Combining It With AI.
Chitra Malode (LinkedIn)
Most QA engineers are learning automation, many are learning AI, but very few are learning how to combine both. Lists 10 AI tools every QA professional should know in 2026: ChatGPT, Claude, Gemini, Perplexity, GitHub Copilot, Cursor, Postman AI, Windsurf, NotebookLM, and Firebase Studio. Proposes a roadmap: Manual Testing to AI Tools to Automation to AI-Powered Testing to Better Quality Engineering.
Quick Links
Claude Hidden Gems: Community Tips for Power Users
r/ClaudeAI (Reddit)
Reddit thread with 160+ comments on Claude power-user workflows. Top consensus: context management is the biggest game-changer. Write plans to plan.md before coding so they survive context compaction. Keep a running decisions.md. Use a custom end-of-session command (/done, /handoff) to create clean handoff prompts. Start sessions with seed docs. Use /goal for autonomous execution, hooks for automation, and orchestrate multiple models (Opus for planning, Sonnet for grunt work). Elite prompting hacks: have Claude interview you instead of dumping context, force it to ask 3 clarifying questions before coding, and set up adversarial sub-agent panels to test code.
AI Rewards Communication Skills, Not Just Tool Access
Tatyana Arbouzova (LinkedIn)
Giving someone an AI tool doesn't mean they know how to use it well. AI is closer to working with a human than a vending machine. The actual skill is the loop: giving context, correcting, re-explaining. If you don't already have strong communication skills or know how to give context clearly, you'll lose the AI game, not because you lack access but because you lack the skill the tool rewards. Three years into LLMs, courses that teach this well barely exist.
My AI Native Engineering Loop
Jesse Merhi (jmerhi.mov)
An OpenClaw maintainer's full AI-native development workflow. The loop: find the problem, brief the agent with full context (MCP, docs), make it grill you with questions before coding (-with-docs), turn the conversation into a spec and tickets, hand off to parallel agent sessions in tmux worktrees, test by talking to it, review it cold with blind subagents (-pr-review-until-clean), then ship. Key insight: cold-context reviews are the most honest because the reviewer has no memory of writing the code. Built tooling for agent worktrees, tmux cockpit management, session search, and review memory.
Our Best Technical Hire Is Not a Developer. She's a QA Engineer.
Alexander Rogovsky (LinkedIn)
A CEO says their best technical hire is a QA engineer named Ksenia, because she understands the product better than the people who built it. Coders focus on their piece, but the person testing everything sees the whole product. In the AI age, when the cost of coding became cheap, QA engineers might just be the smartest people in the room. Comments reinforce: QA is the only team that knows how things should be, how they currently work, and what users actually expect.
Opus 5 Is More Proactive in Testing and Self-Validation Than 4.8
Artem Bondar (LinkedIn)
Opus 5 proactively tests and self-validates without guidance. After completing a feature, it adds test data to the database over MCP, runs functional tests headlessly in the browser, and cleans up afterward, all on its own. It also proactively adds Playwright tests alongside features, including test coverage in the plan. With good rules and skills in place, it requires only minor corrections after code review.
If something in this issue made you think differently about how your team approaches AI in testing, pass it along. The best conversations about AI and QA are happening in Slack channels and stand-ups, not just newsletters.
Have something worth featuring? Reply and send it my way, I read every link.
Thanks for reading,
Butch Mayhew