Sponsorship
Interested in sponsoring AI in QA? Reach out.
AI in QA Review
Butch's Take
If you haven't taken the poll, do that first: What AI tool do you use the most?
This week I've run across a lot of posts blurring two very different things: The difference between "Testing with AI" versus "Testing the AI". I'm mostly talking GenAI and LLMs here.
"Agentic Tester" is the term that muddies it most. Does that mean you're testing agentic workflows, or that you're an agentic tester using AI in your day to day work? Two different jobs. It's worth pulling those apart, because they're not the same job.
Call the first one Testing with AI. You're using AI to assist or augment how you test. Doesn't matter if the thing under test is an AI feature, a traditional web app, a fat client, or a mobile device. The AI is in your toolkit as you test and evaluate the system.
Call the second one Testing the AI or "AI Tester". Now the AI itself is the thing under test. You're validating its output, checking its goodness, poking at whether it actually does what it's supposed to.
I do include both in the newsletter, but most of what I cover lands within the Testing with AI category, and here's why . The systems most testers are actually testing today June 15th 2026 are enterprise applications. Some have AI bolted on now, a summary here, a chatbot there. Very few are AI-first. My hope is the articles I cover will help you during your day to day work, testing big, mostly-deterministic systems that happen to have a little AI baked in.
I spend the majority of my day training large QA teams, on leveraging AI in their work day to day work. If you are on a team interested in leveraging AI tools in your testing work, connect with me on LinkedIn, and let me know.
So a small challenge. If you're not already doing Testing with AI, start. Use it to write deterministic test scripts, or assist with your test planning. Use it where it actually helps, skip it where it doesn't. The only way to learn where that line is, is to use the thing.
Headlines & Launches
The Job Your Boss Thinks You Do Is Disappearing
The Vernon Richard Show (YouTube)
Keith Klain's blog post sparks a deep-dive on the testing work the market is rapidly reassessing. The panel unpacks checking vs testing, why CEOs value what they value, and what happens when AI comes for the execution-heavy work testers have treated as their core identity. Key threads: shared understanding over assumed understanding, shift-left as more than 'test the requirement first', and the skill no one taught testers that now matters most. Features discussion of Maaike's stance on refusing AI entirely, forward-deployed engineers as QE rebranded, and AI summaries as input to other AI.
Looking Good, Testers!
James Thomas (Hiccupps)
James Thomas (Hiccupps) argues that testing is fundamentally about looking, and AI tools sit at the far end of a spectrum that distances testers from source material. He makes a case for intentionally staying close to the product: retyping requirements, exploring manually, building mental models. The friction of direct engagement is where learning happens, and over-delegating to AI risks eroding skills, system knowledge, and the peripheral vision that catches what tools miss. A thoughtful counterweight to the 'automate everything' narrative.
AI Code Generation Makes Manual Exploratory Testing Valuable Again
Daria (LinkedIn)
Daria argues AI-generated code creates a new class of bugs: the code works for its intended purpose but unintentionally breaks shared areas, permissions, UI states, or integrations. Automated tests still pass because they validate expected behavior. The gap is exploratory testing backed by deep system understanding. AI accelerates development and automation while simultaneously increasing the need for critical thinking in QA.
Coding Agents Delete Failing Tests Instead of Fixing Code
Noah Sussman (LinkedIn)
Noah Sussman highlights an ACM report finding that AI coding platforms modify, disable, or delete failing tests rather than fix the underlying bugs. The old warning that you can't write a program to verify another program now has teeth: LLMs actively refuse to determine correctness by making failing tests disappear. If you hand a vibe coding system an acceptance test suite, you cannot assume it will be intact when the system finishes.
AI Agent QA Orchestration Claims Are Mostly Hype
Andrejs Doronins (LinkedIn)
Andrejs Doronins pushes back on the wave of 'I built a QA agent framework' claims, arguing most are just markdown prompt chains that will fall apart past proof-of-concept or burn tokens on poorly written, failing tests. He puts his real AI productivity boost at 20-30% and prefers using AI to improve quality over quantity.
Falling behind on test automation and AI adoption? DevClarity's QA Practice gets your team up to speed fast - with hands-on training, proven workflows, and measurable results within 30 days.
Tools & Frameworks
WebdriverIO MCP v3.6.0: Cloud Device Farms via Natural Language
Vince Graics (LinkedIn)
WebdriverIO MCP v3.6.0 adds BrowserStack, SauceLabs, and TestMu (formerly LambdaTest) support to a single MCP server. AI assistants can now launch real browser and mobile sessions across cloud device farms using natural language. Includes Appium support for native iOS/Android, tunnel support for localhost/staging, and Vibium-compatible traces for every session. The shift: AI agents can now execute and validate on real infrastructure, not just generate code.
Playwright Screencasts and Agentic Receipts
Martin Poole (Blog)
Martin Poole walks through Playwright 1.59's new page.screencast API for producing annotated video evidence of test runs. The key idea: agentic receipts. When an AI agent runs tests via browser.bind(), it can leave behind a screencast with chapter markers, action annotations, and overlays that explain what it did and why. Instead of a wall of text logs, reviewers get a narrated video trail. Practical for anyone running agent-driven testing who needs auditable, reviewable output.
GitHub Copilot Pricing Bump: Users Report 5-10x Cost Increase
Pramod Yadav (LinkedIn)
Pramod Yadav reports GitHub Copilot's new pricing model burns through a month's budget in 3 days, a roughly 10x increase over the old 39 EUR/month with 1500 requests plan. Comments confirm similar experiences across multiple users. Thread explores alternatives including Claude Code, Codex, and open models. Relevant for QA teams budgeting AI tooling: flat subscriptions vs. usage-based billing is now a real decision.
Foundations
Progressive Disclosure: Structure Your AI Agent Orchestration Like a Book
Ivan Davidov (LinkedIn)
Ivan Davidov argues that one giant system prompt causes context rot, where rules near the bottom get applied inconsistently. His solution is progressive disclosure: layer 1 is CLAUDE.md (back cover, ~100 tokens), layer 2 is individual SKILL.md files (chapters, under 5k tokens each), layer 3 is references/ with full examples loaded only when needed. The same pattern Anthropic shipped as an open standard in December 2025.
Techniques & Tutorials
Modern QA in 2026: What Actually Matters Now
Paul Morris (Blog)
Paul Morris argues QA isn't dying, but low-value parts of it are. Manual regression, gatekeeping, and writing test cases nobody reads are being replaced by faster feedback loops, better tooling, and AI handling repetitive work. Modern QA means shifting left (involved before code exists), thinking in systems over test cases, prioritizing feedback speed over coverage metrics, and working alongside engineers instead of after them. The demand for quality is higher than ever, but the approach has changed.
Test Automation Isn’t That Hard: Are You Just Doing It Wrong?
Emily O’Connor (Leeds Testing Atelier)
A talk challenging the common narrative that test automation is inherently difficult, arguing that many teams struggle not because automation is hard but because they’re approaching it wrong. Covers practical insights on strategy, tool selection, and avoiding common anti-patterns.
The Test Suite Was the Incident
Christopher Meiklejohn (Blog)
Christopher Meiklejohn's postmortem of a night where AI-generated test fixtures caused 49 failed CI runs and in wasted costs. The core problem: agents writing tests reuse shared fixture data because it's the cheapest path to green, creating a global mutable state that nobody owns. When one test's fixture change breaks another test's assumptions, every PR pays the toll. The lesson is about test data ownership and isolation, not AI capability.
Agentic Testing and the New Bug Report Workflow
Maaret Pyhäjärvi (LinkedIn)
Maaret Pyhäjärvi describes a shift in how testers handle bugs: report the symptom to an AI agent, get back a test plus fix in a pull request. She also argues that vibe coding has convinced her cross-browser testing matters more, not less, and admits relying on manual 'oh, it's Wednesday, time for Firefox' exploration over automated browser tests for these scenarios.
Hot Take 🔥
Folks that have any of these in their LinkedIn Banner
Agentic SDET | AI-Augmented Test Automator | LLM-Powered Eval Architect | Autonomous Regression Orchestrator | Self-Healing Test Suite Visionary | AI-Native QA | LLM-Powered Test Strategist | RAG-Enhanced Eval Architect | Hallucination-Resistant Assertion Engineer | Synthetic Test Data Specialist | Human-in-the-Loop (Reluctantly) | MCP-Native Orchestrator Reimagining Quality in the Post-AGI SDLC
Research & Data
AI and Testing: A Knowledge Graph Pipeline in Practice – Stories from a Software Tester
Jeff Nyman (Tester Stories)
Jeff Nyman builds a working knowledge graph pipeline that extracts typed RDF triples from scholarly text, constructs a reified graph, and queries it with SPARQL. The pipeline runs end-to-end with a local model (Qwen 2.5 via Ollama) and includes four query types: entity neighborhood, predicate filtering, multi-hop traversal, and confidence diagnostics. The key insight for testers: the architecture makes failure modes visible and debuggable because extraction output is structured, the graph is inspectable, and queries are explicit. A grounded answer stage refuses to confabulate when results are insufficient, which is the system working correctly.
How LLMs Actually Work
0xKato (0xkato.xyz)
0xKato wrote a 26-minute walkthrough of transformer internals without the math. Covers tokenization, embeddings, positional encoding (including RoPE), attention, multi-head attention, the feed-forward network, residual streams, layer normalization, and next-token prediction. Each section has tiny explainers for non-technical readers. Good foundational reference for QA folks who want to understand what the models they're testing actually do under the hood.
Quick Links
Automating Tasks with AI Agents and Evaluating Performance
Tatyana Kazakova (LinkedIn)
Tatyana Kazakova's viral post traces the slippery slope of AI-powered QA: start with a ChatGPT prompt for test cases, then add Cursor for code tracking, Obsidian for memory, Claude Code agents for risk analysis, Playwright agents for speed. Soon you have 20+ agents and you're writing evals for agent managers. A fun mirror for anyone building agentic QA workflows.
A Deterministic Unity Playtester for AI Agents
Lorenzo Nuvoletta (X)
Lorenzo Nuvoletta built a deterministic Unity playtester that his AI agent uses to create, run, and record tests inside the actual editor. He abandoned Computer Use as too slow and unreliable for game testing. The approach runs fast and is repeatable, letting agents self-test game features they build. Interesting crossover: agentic QA expanding beyond web into game development.
agent-browser v0.27.2: Silent Failures Are Now Loud Ones
Chris Tate (X)
agent-browser v0.27.2 converts silent browser automation failures into actionable errors: off-screen clicks auto-scroll, dialogs no longer hang, select errors list valid options, and blocked clicks identify what’s covering them. An alternative to playwright-cli and Playwright MCP for AI agent browser control, with ~150ms speed improvement.
End Every Claude Coding Session With This One Question
Brian Batt (LinkedIn)
Brian Batt's simple hack: end each AI coding session by asking 'what can we add to your memory to make you more effective and use less tokens on these tasks?' Across 35 saved memories, he estimates saving over 1 million tokens by avoiding repeated investigation work. Amir Nathoo adds a related tip: turn recurring CLI mistakes into skills so the agent stops re-verifying usage every time.
QA Engineer Asks: Pivot to Dev or Stick to My Guns?
Chris Zeuch (X)
Chris Zeuch captures the current QA identity crisis: bugs persist despite AI, devs own testing, companies aren't hiring QAs, and the role feels redundant outside FinTech. He's weighing whether to pivot to development or tooling.
If something in this issue made you think differently about how your team approaches AI in testing, pass it along. The best conversations about AI and QA are happening in Slack channels and stand-ups, not just newsletters.
Have something worth featuring? Reply and send it my way, I read every link.
Thanks for reading,
Butch Mayhew