Sponsorship
Packt is running a four hour hands-on workshop on agentic test automation with Playwright on October 21, taught by Ivan Davidov and Debbie O'Brien. The VIP pass includes an e-copy of Hands-On Automated Testing with Playwright, the book I authored (code BU30 takes 30% off either package).
Butch's Take
ThePrimeagen just joined Omarchy Core (DHH's Arch-based Linux distro) to lead Agentic QA. He's spent the past month building Oligarchy, a harness that boots Omarchy in QEMU virtual machines. Agents then drive the desktop the way a person would: keystrokes, mouse clicks, screenshots, and a record of whether each step did what it was supposed to.
The big question for me is how much of what QA professionals have spent years learning will make it into these agentic test automation tools. Are the agents verifying, or testing? Running checks, or exploring? ThePrimeagen's approach has an agent driving the testing, so I want to see how those tests get captured and what the tool ends up looking like.
Here's a deeply technical developer stepping into software testing, and his first move is building a harness to test an entire operating system. While I don't see devs moving into testing as a trend yet, I do find this interesting. As many developers are banging the drum "the coding problem is solved", I could see more and developers shifting to other interesting problems. Testing, evaluations, verification, infrastructure, etc... There are a lot of strong critical thinkers among developers, and some of them are going to start picking up testing work.
Headlines & Launches
ThePrimeagen Joins Omarchy Core to Lead Agentic QA
David Heinemeier Hansson (omarchy.org)
DHH announces ThePrimeagen leading Agentic QA for the Omarchy Linux distribution. His Oligarchy harness boots the OS in QEMU VMs where agents send keystrokes, click the mouse, screenshot, and record results, each test starting from a freshly minted disk. The stated rationale: no human team could click through every release across platforms.
Your Agent-Written Tests Aren't Real Tests
stack72 (stack72.dev)
The argument: when an agent writes both the code and its tests, both encode the same interpretation, so the tests are specification, not verification. Cites a study of 86,000 agent test patches where 80.2 percent had weak or missing oracles, and finds real verification only where the agent cannot rewrite the test to reach green.
The Stoic Tester: AI Is Forcing Us to Ask Who We Are
Stephen Platten (LinkedIn)
Platten argues AI is a philosophical shift, not just a technological one, and walks through what it destabilizes: experience (processing it is not having it), identity (the task was never the value; understanding why we needed the test was), professional belonging (where will people learn the craft of quality), and trust in evidence itself. His closing question for testers: not what can the machine do, but what do we want to remain responsible for?
The Test Suite Is the New Code Review
Allen Hutchison (allen.hutchison.org)
Hutchison's Vycari series continues: with agents opening pull requests and Greptile reviewing in minutes, the test suite became the shipping gate, so he restructured around it. A 13-minute suite backs a merge queue of agent PRs, overnight tiering can't stand in front of 15-minute deploy cycles, and slices pulled into their own repositories clear CI in two minutes. The gating factor moved from human judgment to compute time, the one delay engineering can attack.
Agent Breaker: Lakera's Arcade of Hackable AI Agents
Lakera (Lakera)
Lakera, the team behind Gandalf, launched Agent Breaker: a free testing playground of ten realistic GenAI apps, from a code-review bot to a legal assistant, each hackable through prompt attacks. Objectives mirror real incidents: extract system prompts, exfiltrate data, weaponize agents against other users. Attacks score 0-100, 75+ unlocks the next level, and indirect vectors include poisoned PDFs, MCP server descriptions, webpages, and chatbot memory logs.
Tools & Frameworks
Jevify: An Agent Prompt for Finding Jev Opportunities in Your Project
Ryan A (GitHub)
A reusable prompt you paste into your coding agent to investigate where Jev, TypeSafe's structured decision model, fits an existing project. Jev returns choices, rubric scores, and yes/no probabilities instead of generated text, and the prompt bakes in evaluation discipline: baselines, calibration checks, adversarial inputs, and go/no-go criteria.
Agent Skills for Building Product-Specific AI Evals
Hamel Husain & Shreya Shankar (GitHub)
Eight agent skills from the team behind the AI Evals course that guide a coding agent to help build product-specific evals. The centerpiece, error-discovery, clusters traces, samples for diversity, builds a review app, and turns free-text notes into failure modes before you write any eval. Also includes judge calibration against human labels.
Mobile-Use: AI Agents That Drive Real Android and iOS Apps
Minitap (GitHub)
Minitap's open-source agent drives real Android devices and iOS simulators from natural language commands, extracting structured data from any app. It claims first place on AndroidWorld at 100 percent via task decomposition, runs on pluggable providers, and ships with a research paper on the multi-agent architecture. Games without accessibility trees remain out of reach.
MyA11yReport MCP: Accessibility Audits for Your Coding Agent
MyA11yReport (mya11y.report)
A local MCP server that gives a coding agent real accessibility audit tools: it drives Chromium via Playwright, runs axe-core, and reports landmarks, tab order, image alt text, and WCAG 2.2 contrast. Fully local with no API key, and the audit guide flags which checks a human must still verify.
Foundations
No items yet.
Techniques & Tutorials
Six Boundaries to Test Before Trusting an AI Agent
Pentesty (Pentesty)
Pentesty turns OpenAI's September misalignment disclosures into a six-boundary agent security checklist: identity and scope, tool permissions, data destinations, human approval, memory integrity, and evidence integrity. Each boundary pairs a controlled test with the evidence to retain, and the core advice is tester-speak: treat retrieved content as untrusted input, verify server-enforced access separately from prompt instructions, and never accept the agent's own account of its work as proof.
A 7-Point AI Governance Framework for Quality Professionals
Beyond Quality Podcast (YouTube)
The Beyond Quality podcast (Vitaly and Anupam) hosts Brijesh Deb on why AI makes governance gaps too big to ignore, opening with the agent that deleted a production database while hunting an API key. Deb offers a 7-point AI governance framework covering intent drift, autonomy boundaries, memory integrity, permission accumulation, evidence strength, realistic human review, and monitoring of living workflows.
Jev Drove 10 FX Trading E2E Tests, 5 Cents Total, Zero Selectors
Amit Rawat (LinkedIn)
Rawat ran 10 RFQ end-to-end cases against his open-source FX trading platform with TypeSafe's Jev making every browser decision: pair, notional, tenor, quote requests, best-price acceptance. Only hand-written code was the blotter assertion. 10/10 passed in 81 seconds across 58 decisions, costing 5 cents total. When blocked, Jev reported low-probability blocked instead of claiming success, and rerunning with a local 7B doing the typing kept the suite cloud-free.
How I Take AI Agents From Demo to Production in Financial Services
Anantha Subramanya (ananthasubramanya.com) JACLYN LILLY
A quality engineering lead in financial services lays out an evaluation strategy for regulated agentic AI: versioned datasets built from real traffic with golden, regression, and adversarial layers, trajectory-level scoring because a right answer via the wrong path is a compliance issue, and release gates where a pass-rate drop blocks the build. Variance is the signal: a task passing three of five runs is a latent incident.
Research & Data
How to Evaluate LLMs Before Production
Mariko Wakabayashi, Zixiao Chen (The GitHub Blog)
GitHub's applied scientists share evaluation practices from cutting secret-scanning false positives. Metrics are tiered into primary outcome, safety constraint, and operational guardrails, with recall as a guardrail precision gains cannot violate. Offline evaluation runs like integration testing with versioned prompts and one-variable-at-a-time baselines, production labels are treated as signals rather than truth, and error analysis assigns each failure to model, prompt, input, pipeline, dataset, or label.
Novelty Laundering: When Research Discovers System Testing
Keith Klain (LinkedIn)
Klain reacts to an empirical study of 157 open-source agent projects: 137 had basic functional tests, 95 security or safety tests, but only eight had explicit prompt-injection, adversarial, jailbreak, or red-team paths. The paper's prescription of covering the complete model-to-action workflow across execution surfaces and permission chains earns his coinage 'novelty laundering' — old ideas, rebranded, history ignored. Testers already know all of this as system testing.
Hot Take 🔥
Rauch: The Thoroughness of Agent QA Is Unrivaled
Guillermo Rauch (LinkedIn)
Rauch (Vercel CEO) reports pointing out a mobile rendering bug and watching an agent reproduce, fix, deploy to an ephemeral Vercel environment, and verify against an iPhone simulator. 'Humans could simply never match this level of intensity.' The comment thread earns the click: a tester pushes back that thoroughness is one aspect of testing, and an Angular nine-version migration run overnight with pixel-by-pixel visual comparison against screen captures of the old app.
Quick Links
Playwright CLI + Jev vs Playwright MCP: 98 Percent Cheaper, Twice as Fast
Filip Hric (LinkedIn)
Hric pits Playwright CLI plus Jev against Playwright MCP on a fuzzy Trello instruction, create new board, list and card. Handing Jev a Playwright snapshot and letting it pick the element came out 98 percent cheaper and twice as fast than the snapshot-and-tool-call MCP loop.
53 Bugs From 4 Months of Running an Agent Harness in Production
Michael Piccolo (GitHub)
A team running an agent harness in production for four months catalogs 53 failures, nearly every one linked to the fix. The list doubles as a test-ideas backlog for anyone testing agents: poison messages and redelivery loops, cut streams parsed as success, silent subagent deaths, compaction returning empty context, deny rules failing open, and confirm-before-restarting living in the prompt instead of the code.
Build One Small AI Agent for a Repetitive Daily Task
Suraj Narayan (LinkedIn)
Narayan's advice: every QA engineer should build one small agent for a repetitive daily task, not a grand autonomous testing system. He lists nine candidates, from requirement review and API contract agents to failure triage, flaky-test analysis, and regression selection, with the caveat that an AI-generated test can look correct while validating the wrong business behavior.
If something in this issue made you think differently about how your team approaches AI in testing, pass it along. The best conversations about AI and QA are happening in Slack channels and stand-ups, not just newsletters.
Have something worth featuring? Reply and send it my way, I read every link.
Thanks for reading,
Butch Mayhew
