Sponsorship

Interested in sponsoring AI in QA? Reach out.

AI in QA Review

We're live again this week with Issue #21, if you want to drop us a question or upvote an existing one we answer these live during the review: Ask a Question.

Butch's Take

This past week, Bob Martin, author of Clean Code, shared that his current strategy is to not read any of the code written by his agents. This is a very interesting take, relying on what he calls "extreme constraints":

Unit tests, Gherkin tests, QA procedures, quality metrics, mutation testing, test coverage, and a plethora of others. In the end, I have very high confidence in the code they produce because they've had to run the gauntlet of all of my constraints and tests.

While I like the spirit behind this idea, getting all the right "checks" in place to validate the software is doing what it is supposed to do, this seems like a very risky place to be. What about all the non-functional areas that aren't easy to put constraints or tests around? What if your tests are checking the wrong things, are we at least reading this code if generated by agents? There are 10 more roads we could go down asking more "How, What, and Why" questions but the biggest thing that makes me think this isn't the right path is who is responsible for the code that the agents wrote? I don't see a CTO firing an agent for shipping a feature that caused a lot of harm or loss for the company.

Where this idea thrives is on personal projects, internal tools, prototypes, simple software, or other projects which have a low footprint, risk factor, and can be easily verified through automated checks. Think about that tool that does 1 or 2 things and that is it. Shipping Enterprise level software in this fashion just seems irresponsible.

Context matters. I've spent a lot of time building little tools and utilities, not reading any of the code, just reading the high level summary of the changes, with success. Would I ever attempt this on an application with multiple developers, with paying customers, in a production environment? I don't think so. I'm at least peeking at the code to make my own assessment because in the end I'm responsible for the code my agents write.

Headlines & Launches

I Was an AI Sceptic (And I've Got the Old Blog Posts to Prove It)
Callum Akehurst-Ryan (cakehurstryan.com)
Honest reassessment from a tester who was an AI sceptic in 2023 and wrote about it. Key realization: garbage in, garbage out is not a reason to doubt AI, it is the best news for testers. If output quality depends on input quality, and putting good thinking into messy problems is literally the job of a tester, then AI rewards testing skill rather than replacing it. Answers each of his old objections with what he learned since.

Verification Is Quietly Becoming the Most Valuable Skill in AI
Amit Rawat (LinkedIn)
Amit Rawat pulls a quote from Boris Cherny, engineer behind Claude Code: verification is probably the single most important thing people do not get right. Prompt engineering and context engineering come and go, but knowing whether output is actually correct remains a human job. Generation is cheap; defining what good looks like and what the test actually is becomes the real moat, and it is the exact muscle QA has been building for years.

Three Companies Paying Up to $250K for Software Quality Engineers
Wei-Wei Wu (LinkedIn)
WRITER, Fieldguide, and Mandolin are paying up to 250K for software quality engineers who own test frameworks, CI gates, and ship decisions. None are AI labs; they put AI in front of auditors, hospitals, and enterprise buyers where wrong answers are expensive. The post cites Anthropic data that Claude now writes 80% of their merged code, but the 8x productivity figure almost certainly overstates real gains because lines measure quantity, not whether code works. The bottleneck is moving to whoever can tell whether the code works, and pay is climbing because producing code got cheap but proving it works got harder.

QA, Shift-Left, and the Struggles of Cognitive Load
r/QualityAssurance (Reddit)  VIA Volha Baban (LinkedIn)
A 20-year QA veteran approaching 50 is burning out because their company implemented shift-left testing and doubled their workload without adding resources. They are now doing test case creation before seeing the product, risk analysis, design reviews, heavy documentation, early dev team involvement, ongoing testing, and constant task switching. Senior QA engineers responded unanimously: the company confused shift-left with QA does everything earlier. True shift-left redistributes testing across the team. Advice includes reframing as a quality coach, forcing leadership to make trade-offs, writing detailed test cases after seeing the product, and leveraging AI for routine work.

Tools & Frameworks

Testing AI Book Now Available as Plugin for AI Coding Agents
Jason Arbon (LinkedIn)
Jason Arbon releases the Testing AI book as a plugin for Claude, ChatGPT, and other coding agents. Add the plugin, point it at your project, and it recommends ways to improve validation, testing, and confidence. Goal: make AI-first QA accessible directly inside your existing coding workflow. Beta available now.

Two AI Agents for Risk-Based Quality Engineering
Taras Mysko (LinkedIn)
Built two QE agents: an Automation Agent that reads tickets, checks code diffs, proposes test cases, writes automation, runs it, and self-reviews; and a Sprint and Impact Agent that pulls code changes, maps impact, handles test planning, and helps with go/no-go decisions. AI handles 80% of the heavy lifting, humans still own the tricky 20% of edge cases and domain context.

Bruno v4 Adds AI: Generate Tests, Docs, and Scripts Inline
Bruno (LinkedIn)
Bruno, the open-source API client, ships AI in v4. Generate tests, docs, and scripts directly from your collections. Inline autocomplete as you type. Bring your own API key (OpenAI, Anthropic, or any OpenAI-compatible endpoint), kept 100% local. No data leaves your machine.

GPT 5.6 Sol: The Best Advocate for Open Models
Dan Harper (LinkedIn)
Dan Harper's candid field report on GPT 5.6 Sol: it over-engineers, drifts repeatedly, ignores corrections, and takes roughly 8x more iterations than GPT 5.5 to reach a workable result. A planned performance fix with clear examples and documentation still resulted in 48 incorrect changes within an hour. Planning is the most broken feature: ask one clarifying question and the model does a complete 180 on its own plan. His takeaway: GPT 5.6 is the best advocate for open, multi-model workflows he has seen in a while.

Trowser: An AI-Enabled Browser Built for Testers
Alan Richardson / Rikard Edgren (Evil Tester)
Alan Richardson interviews Rikard Edgren about Trowser, a free Chromium-based browser built specifically for software testers. Wraps Chromium in C# with integrated testing tools: accessibility checks via axe-core, network interception, script injection, customizable headers, and a rich REST API that lets an LLM drive the browser alongside the human tester. Rikard found AI-driven exploratory testing surprisingly strong, especially for security testing. Key insight: the tools we use shape our thinking about what testing is, which is why a browser purpose-built for testers could shift industry approaches.

Foundations

Quality Isn't an Activity, It's a System
Dan Ashby (LinkedIn)
Part 1 of 4 series rethinking the quality operating model for AI-native engineering. Argues quality has outgrown the bolted-on testing team model. Quality is an emergent property of a distributed system (devs, testers, AI agents, tools) that needs a coordination layer. AI's real role is connective tissue between tools that were never designed to talk to each other. Testers evolve into quality system designers, not execution bots. Includes a 5-layer model for the new quality operating system and practical examples from SmartBear's BearQ.

Testable Diagnostics for Comprehension Debt
Dr. Cat Hicks (Fight for the Human)
Pushes past the comprehension debt metaphor to testable diagnostics. Key insight: 96% of developers don't fully trust AI code correctness, but 48% still commit it without checking. Distrust alone doesn't predict different behavior, which means the real driver isn't individual judgment failure but organizational conditions. Introduces Overproduction Pressure as a measurable condition that makes understanding-building work harder to justify and slower to practice. Argues comprehension debt is a symptom, not a cause, and measuring the conditions underneath tells you which intervention will actually move the needle.

Techniques & Tutorials

What Verification Actually Looks Like When Building With AI Agents
Biju Tholath (LinkedIn)
Field report from building an AI insurance platform. AI agent reported a phase as complete and tested. pytest collected zero items: no tests existed yet, by design. A health endpoint redirect was blamed on Django locale middleware that does not exist in the project. Both catches only happened because the developer checked the actual source, ran the actual commands, and queried the actual database instead of trusting the agent's summary. Security-critical code held up under line-by-line scrutiny, but the lesson stands: building with AI means verifying every claim against reality.

Research & Data

Recall, Relevancy, and Richer Evaluation
Jeff Nyman (Tester Stories)
Adds three RAG evaluation metrics to the diagnostic toolkit: Contextual Recall (did the retriever find all needed chunks?), Contextual Relevancy (how much noise was in the retrieved set?), and G-Eval (custom criteria via a judge model). Uses a philosophical essay as a stress test where vocabulary overlap across sections creates a deliberate trap for semantic retrievers. Key finding: for argumentative prose, Contextual Relevancy scores should be read as relative signals, not absolute thresholds. G-Eval catches failures that RAG-specific metrics cannot see, like shifting an exploratory argument into an assertive register.

Hot Take 🔥

Hot Take: AI Code Review > Human Code Review
Cory House (X)
Cory House's spicy take: most devs are too lazy, too busy, or too incompetent to do a good code review, so the average AI code review is already better than the average human review. And AI is improving every day. Follow-up: AI can simultaneously review security, accessibility, and performance, consistently honors instructions, never gets tired, and never just says looks good to me.

Run Tests Using Natural Language with k6 Agentic Testing
Grafana (YouTube)
Grafana demonstrates k6 Agentic Testing, letting you write and run load tests using natural language prompts instead of scripting. Part of k6's move toward AI-assisted performance testing.

Running a 28.9M Parameter LLM on an $8 Microcontroller
Nick Bild (Hackster.io)
Someone got a 28.9 million parameter LLM running entirely on an ESP32-S3 with 512KB SRAM and 8MB PSRAM. The trick: Per-Layer Embeddings borrowed from Google's Gemma models, storing 25M parameters in flash as a lookup table and reading only ~450 bytes per token during inference. After 4-bit quantization the model is just 14.9MB. Generates text at 9.5 tokens per second on an $8 board. Trained on TinyStories so it writes short simple stories, but still impressive as the first known adaptation of Per-Layer Embeddings to hardware this constrained.

GitHub Stacked Pull Requests
GitHub (GitHub Docs)  VIA Deepak Kamboj(LinkedIn)
GitHub's stacked pull requests let you break large changes into a chain of smaller, dependent PRs you can review and merge independently. Includes the gh stack CLI extension, org rollout guide, CI optimization tips, and rebasing tools. Useful for teams shipping large features who want smaller, more reviewable PRs.

If something in this issue made you think differently about how your team approaches AI in testing, pass it along. The best conversations about AI and QA are happening in Slack channels and stand-ups, not just newsletters.

Have something worth featuring? Reply and send it my way, I read every link.

Thanks for reading,
Butch Mayhew