Sponsorship
Interested in sponsoring AI in QA? Reach out.
Butch's Take
"Confidence is the new coverage" runs through a few items in this issue. Jason Arbon's book, the manifesto, Janna Loeffler's post all point the same direction: pass rates and coverage percentages don't answer the question anyone actually cares about, which is whether this thing works in production.
I agree with the direction. I'm less sold on where it gets applied.
There's a growing vocabulary around confidence engineering, and I think we're at risk of over-pivoting on it. Confidence is a feeling. A team can feel great about a release and still take down production, especially if we're applying confidence to the wrong systems.
Here's how I think about it. When you're testing an AI feature, there's no correct answer to assert against. The same prompt gives you a different response tomorrow. You're judging whether output is good enough, which means evals, scoring, sampling, thresholds, and a human deciding where the bar sits. Confidence engineering should be what we reach for there.
Traditional software is different. Given the same input, you get the same output, every time. Leveraging deterministic test automation scripts, we can know with certainty when the system has changed. Every check you write is a small bet about what could break, and every run tells you whether anything changed. That's information you didn't have before. Confidence comes out the other side as a byproduct, based on what features and tests are covered.
My thought around this is that over-correcting and applying probabilistic evaluation to a system with deterministic outputs actually leaves me with less confidence. The same reason I haven't deleted my test automation suites and handed the job to an AI tool with /goal find all the bugs. That's a useful thing to run, and I do run it. I'm just not shipping to production when the AI reports back after it's tests are complete.
Match the method to the system. Use evals where you have to, keep your assertions where you can.
Headlines & Launches
Engineering Confidence in Non-Deterministic Systems by Jason Arbon
Jason Arbon (Amazon)
New book from Jason Arbon (Microsoft, Google, test.ai) on testing AI systems with confidence. Covers evals, rubrics, release gates, LLM judges, confidence intervals, RAG testing, agent testing, prompt injection defenses, and production monitoring. Practical guide to deciding whether AI behavior is trustworthy enough to ship. Connects to the Confidence Engineering manifesto.
Rethinking QE Leadership Compensation in the AI Era
Janna Loeffler (LinkedIn)
Argues that QE leadership roles need repricing. The traditional assumption that testing leaders are 'less technical' is outdated. The real challenge is determining whether you can trust what ships, not running test suites. Introduces Confidence Engineering: probabilistic evaluation over binary pass/fail, risk mitigation over verification, system-level thinking. References the Manifesto for Confidence Engineering at opentest.ai/manifesto.
Manifesto for Confidence Engineering
opentest.ai (opentest.ai)
A manifesto arguing for a shift from binary pass/fail testing to probabilistic confidence evaluation. Six core values: probabilistic evaluation over binary assertion, human ingenuity over human process, application of AI over AI resistance, mitigating risk over verification, understanding complexity over isolated functionality, and system-level thinking over component-level thinking. Open for public signatures.
QA and AI: Framing the Wrong Question
Daria (LinkedIn)
Argues the real question isn't how QA adapts to AI, but whether you can create clarity before AI creates code. Shift-left thinking, long valuable before AI, now prevents unclarity from becoming working code. AI is only as good as the context it receives: unclear requirements produce tests for the wrong behavior. The QA engineers who thrive will combine AI with strong QA thinking, not just learn to use AI tools.
Tools & Frameworks
Spend Fewer Tokens Without Using Your AI Agent Less
Bruno Guidolim (LinkedIn)
Argues that token savings come from amortizing knowledge across sessions, not just compressing prompts. Four-step loop: search what the agent already knows before opening files, work normally, capture only the verdict at session end, and index it. A colleague cut the same prompt from 16k to 1.5k tokens and 4.5 min to 30 seconds. Links to mcs-cli/memory, an open-source Claude Code hook.
IA-QA CLI - 148 LLM Testing Primitives in Your Terminal
Jean-Christophe Jamet (LinkedIn)
CLI tool with 148 testing primitives for LLM stacks: prompt-injection scans, PII detection, hallucination checks, semantic evaluation, RAG auditing, token counting, and cost analysis. Exposed as an MCP server for AI agents and a zero-dependency npm CLI (@ia-qa/cli). BYOK for LLM-powered tools, key-free for the rest. CI-ready, pipeable output.
AI Agent That Writes Playwright BDD Suites from Jira Stories
Alan Khadir (LinkedIn)
Built an AI agent that reads a Jira story, browses the live app, validates the user journey, identifies locators (adding data-testid attributes if missing), generates a full Playwright BDD suite (feature files, step definitions, page objects) in TypeScript, runs the tests, and self-heals on failures. Open source: github.com/alan-Khadir/jira-to-playwright-agent.
Veta - Autonomous AI Agent Swarm for Android Testing
Vip3r-MC (GitHub)
Autonomous AI agent swarm for Android and mobile web testing. Sub-agent architecture (planner, executor, verifier, reporter) drives observe-decide-act loops against containerized Android instances. Accepts APKs or URLs plus plain-English task descriptions, returns pass/fail verdicts with full action traces and video artifacts. Runs 100% on AMD GPUs via Fireworks AI or self-hosted vLLM/ROCm.
Foundations
AI Agents for Beginners - 18 Lesson Course
Microsoft (GitHub)
Free 18-lesson course from Microsoft covering AI agent fundamentals: intro to agents, agentic frameworks, design patterns, tool use, agentic RAG, trustworthy agents, planning, multi-agent orchestration, metacognition, production deployment, agentic protocols (MCP, A2A), context engineering, agent memory, browser use, scaling, local agents, and security. Includes Python code samples using Microsoft Agent Framework with Foundry Agent Service. 70k stars on GitHub.
Maintainability Sensors for Coding Agents
Birgitta Böckeler (martinfowler.com)
Explores using computational and inferential sensors to help AI coding agents maintain code quality. Covers static analysis (ESLint, dependency-cruiser), coupling metrics, AI modularity reviews, and the test suite as a regression sensor. Key insight for testers: mutation testing becomes critical when AI generates tests without human review, since coverage alone masks assertion gaps. Acceptance tests inflate coverage without strong assertions.
Transformer Explainer - Visualizing How LLMs Work
Polo Club (poloclub.github.io)
Interactive visualization of how transformer models work, from tokenization through self-attention to next-token prediction. Powered by GPT-2 (124M parameters) running in your browser. Great foundational resource for understanding what happens under the hood when an LLM generates text, covering embeddings, attention mechanisms, and the full transformer architecture step by step.
Techniques & Tutorials
Evals Are Just QA for AI
Amit Rawat (The Agentic Engineer)
Argues that AI evals are just QA with new vocabulary: task=grader=harness maps directly to arrange=act=assert. Covers evaluating agent skills (trigger rates, ablation, plan-checks vs trigger-checks) and RAG knowledge bases (split retrieval scoring from generation judging). Includes a dilution experiment showing retrieval quality degrades silently as knowledge bases grow. Open-source companion repo with reproducible harness.
Agentic Testing: Where Agents Fit in the E2E Testing Stack
Sergii Gorbachov (Slack Engineering) via LinkedIn
Slack ran 200+ agentic E2E workflows comparing Playwright MCP, Playwright CLI, and AI-generated Playwright tests. Key finding: agents verify goals, tests enforce journeys. Playwright MCP hit 0% failure on simple flows, generated tests were fastest but degraded on complex flows (48% failure). Cost ran -30 per run, driven by context retransmission, not model reasoning. Conclusion: agentic testing adds an exploratory layer on top of deterministic tests, not a replacement.
35-Hour Autonomous Test Coverage Prompt
Dan Harper (LinkedIn)
Shares a goal-based prompt that ran Claude Code autonomously for 35 hours to reach 90% test coverage, finding and fixing 20+ bugs along the way. The prompt enforces strict criteria: 90% lines and branches, no skipped tests, 200ms max per test, black-box testing, business-logic naming, and refactoring where needed. Commenters raise valid concerns about codifying existing bugs as correct behavior and whether coverage equals confidence.
Research & Data
AI and Testing: Testing the Yes-Man in Your Pocket
Jeff Nyman (Tester Stories)
Examines social sycophancy in AI chatbots through a Stanford study across 11 models. Models endorsed users' harmful actions 50% more often than human observers. Covers the LLM-as-judge methodology, validation via human annotators (Cohen's Kappa), and the perverse incentive: users preferred the sycophantic AI that degraded their social behavior. Argues testers must measure downstream psychological impacts, not just satisfaction scores.
Hot Take 🔥
Classic QA Tools Are Becoming Legacy
Ruslan Peyter (LinkedIn)
Argues that Selenium, Appium, and the classic QA automation stack are being displaced by Playwright, Maestro, and AI-assisted workflows. Covers the practical pain points of legacy tools (brittle selectors, slow onboarding, flaky CI) and the tradeoffs of modern AI-augmented alternatives. Honest pros and cons for both camps, with a call to experiment rather than default to habit.
Quick Links
CraftDriver AI Agent Guide
Dimitar Topuzov (CraftDriver)
CraftDriver ships agent-facing tooling: CLI with persistent daemon, MCP server with compact accessibility snapshots, bootstrap files for Claude/Cursor/Copilot/Gemini, and a tiered skill pack. Agents can drive browsers via shell commands or typed MCP tools with auto-waiting and semantic selectors.
Claude Code Confidence Bars Tip
Daniel Shanklin (LinkedIn)
Simple prompt hack: ask Claude Code to add ASCII confidence bars before any multiple-choice questions. Gives a quick visual of what the model is leaning toward. Stored in memories so it persists. Commenters rightly note that self-reported confidence is not calibrated confidence, but it adds a useful signal for reading model output.
Mutation Testing Skill for Claude Code (Stryker)
citypaul (GitHub)
Ready-to-use Claude Code skill for mutation testing with Stryker. Drop it into .claude/skills/ and your agent can set up Stryker, run full-project or diff-against-main mutation runs, triage surviving mutants, and strengthen weak tests. Integrates into a RED-GREEN-MUTATE-KILL-REFACTOR TDD cycle. Includes mutator rules, CI gate guidance, and incremental mode support.
If something in this issue made you think differently about how your team approaches AI in testing, pass it along. The best conversations about AI and QA are happening in Slack channels and stand-ups, not just newsletters.
Have something worth featuring? Reply and send it my way, I read every link.
Thanks for reading,
Butch Mayhew