Sponsorship

Interested in sponsoring AI in QA? Reach out.

Butch's Take

TypeSafe came out of stealth last week with Jev, a System One model . The simplest way to think about it: it's an LLM that can only answer multiple choice. You hand it some state and a set of typed questions, and it hands back one of three things: a category, a score on a scale you define, or a 0-to-1 probability that a statement is true. No explanation, no tool calls. It can't generate text, can't do arithmetic, and can't reliably order dates. What it does is take messy input and return a bounded answer, very quickly and cheaply. TypeSafe's published numbers are 70 to 500ms per call at $0.042 per million input tokens with output free.

Nate on X calls this semi-determinism. You can define the possible outcomes up front, but choosing between them takes real understanding of the situation. Jev is the shim between a messy input and a fixed list of outputs.

That gives us three kinds of tools to now work with instead of two.

  • Code calculates, executes, and enforces the rules.
  • LLMs write, reason through exceptions, and explain themselves.
  • Classifiers (System One model) sit in the middle applying bounded judgment, cheap enough to run on every item instead of a sample.

The temptation in QA is to point this at assertions. I'd resist that, at least to start. Your assertions should stay deterministic. Where Jev actually fits is the judgment work you're already doing by hand every morning: triage. Every failed CI run ends with a human reading a stack trace and deciding "app bug, test bug, test data, environment, etc." Within an hour I was able to stand up a Playwright triage reporter that would take the failure output and run it through Jev. I provided a list of choice names with descriptions that would get passed in along with the test file, error message, and stack trace. The hard part now is defining the buckets and providing a good description that gets passed into the model with the context. This is what will separate Jev-Slop from Jev-Gold.

Headlines & Launches

How Anthropic Scaled Test Impact Analysis as Agentic Coding Strained CI
Anthropic (claude.com)
Anthropic says engineers ship 8x the code per quarter, Claude writes 80% of it, and tests grew 10x, driving a 25x jump in CI jobs. Their test impact analysis service survived three patches (bigger machine, sharding, daily restarts) that bought 70 days, 29 days, and under a day. The redesign: stateless workers plus a journal, built by one engineer in three weeks. Advice: design v0 for 10-20x scale and assume 25x within two quarters.

What Have You Built With AI? Is Becoming a Hiring Question
Chris Kenst (kenst.com)
Kenst hit 'What have you built with AI?' on a job app and had no good answer. Now his employer, FreightPOP, asks engineering candidates which AI tools they use and how they moved their productivity. The signal being measured is curiosity: what you attempted, why, what failed, what you learned. His advice to testers: build something small before you need the answer, because building puts you where quality decisions get made.

Adversarial Browser Testing at Parallel Scale, for Pennies a Release
Rafal Wilinski (X)
Wilinski (founding engineer at Runlayer, previously Zapier's AI agents lead) credits TypeSafe AI's Jev with a massively parallel browser-based adversarial suite that tries to break every release, for pennies. The attached video shows roughly 48 simultaneous browser sessions hammering forms, charts, and tables. Traction suggests the message landed: 339k views and 4,100 bookmarks in a day.

Tools & Frameworks

AutoMancer: Modern Test Automation for Legacy Windows Apps
Paul Turchinetz (GitHub)
New C# engine and CLI for a problem the web world forgot: automating legacy Windows desktop apps, where WinAppDriver has sat unmaintained for years. Locators resolve through a fallback chain (UIA3, UIA2, raw Win32 enumeration) so weak UIA support stops breaking tests, and every action is async poll-until-true like Playwright. Each CLI command is text-in/text-out, the exact shape of an LLM tool call, making it AI-native for RPA. This tool is like playwright-cli/agent-browser for windows applications.

Tencent's BrowserSkill Lets Agents Borrow Your Logged-In Browser
Tencent (GitHub)
Tencent open-sourced BrowserSkill, a CLI plus browser extension that lets any shell-capable agent (Claude Code, Codex, Cursor, OpenClaw) work in your logged-in browser. Agents must borrow a tab, run tasks in a separate visible Agent Window, and return the tab when done, leaving the rest of the browser untouched. Captchas and logins trigger built-in human takeover.

Datamimic: Model-Driven Synthetic Test Data With an Agent Contract
rapiddweller (GitHub)
Datamimic Community Edition generates deterministic synthetic datasets and PII-aware pseudonymized QA exports across PostgreSQL, MySQL, Oracle, MongoDB, CSV, and more, driven by model files rather than source data. The notable addition is the agent contract: an AGENTS.md workflow where AI agents author models as JSON, submit them to a scaffold command, and declare an expectation per requirement, stopping only at verified=true. MIT-licensed; enterprise tier adds PII scanning, governance, and multi-system runs.

Convoy: Selector-Free E2E Testing Powered by TypeSafe's Jev
Gokul Nair (GitHub)
Nair's Convoy runs plain-language end-to-end tests across iOS, Android, and web: it dumps the live accessibility tree, normalizes it to a platform-agnostic control table, and asks TypeSafe's Jev model which control matches the stated intent before tapping. Gates return pass, ambiguous (rephrase the step), or not found (possibly a product bug). No selectors, element IDs, or explicit waits, and one test file targets all three platforms.

Sorify: Rakuten's Agent-Native Browser Testing Platform
rakutentech (GitHub)
Rakuten open-sourced Sorify, a browser testing and automation platform built for AI agents: Claude Code and Codex plugins, an MCP server for low-token control of suites, runs, and cookies, Playwright execution, scheduled runs, and bi-directional webhooks that trigger GitHub Actions. A Chrome extension records real sessions for AI to turn into tests, with AI healing and failure investigation built in.

Atlassian Brings BearQ Into Jira as an Assignable Testing Agent
Inside Atlassian (Atlassian)
Atlassian announced BearQ for Jira on its Marketplace: an assignable AI agent that turns Jira work items into living test context. Assign a ticket once code is ready, and BearQ parses acceptance criteria and comments, authors test cases, explores the app in a browser, and files regressions back into Jira with reproduction steps. Tests adapt as the application evolves, with guardrails set by the team.

Techniques & Tutorials

The Problem With Your AI Tests
Michael Bromley (michaelbromley.co.uk)
Michael Bromley (Vendure) catalogs how AI-generated tests fake confidence: tautologies that assert their own setup, weak assertions that pass for wrong values, happy-path-only coverage, and conditional tests designed to pass no matter what the code does. His fix is a ruthless sub-agent review pass that hunts exactly these patterns, which caught issues in 30% of PRs. Mandating 'test everything' creates volume, not confidence.

Do Agent Skills Help? A 72-Trial Controlled Evaluation
Karl Weinmeister (LinkedIn)
Weinmeister skipped the vibe check and ran 72 controlled trials: Claude Code, 12 Google Cloud scenarios, three attempts each with and without the Google Cloud Developer Plugin, graded by an LLM judge via NVIDIA SkillEvaluator. Skills hit 1.00 correctness and effectiveness vs baseline shell-thrashing (0.00 action efficiency), 100% Pass@3 vs 91.7%, positive lift in 12 of 12 scenarios (p<0.001). The security detail matters: unskilled agents reached for service-account JSON keys; skilled ones used least-privilege identities.

Airbnb Migrated 3.5K Test Files to RTL With LLMs in Six Weeks
Charles Covey-Brandt (Airbnb Engineering)
Covey-Brandt details Airbnb's first large-scale LLM code migration: nearly 3,500 Enzyme test files converted to React Testing Library in six weeks, versus an estimated 18 months by hand. The pipeline ran per-file state-machine steps with validation gates, dynamic-prompt retry loops, and prompts swollen to 40-100k tokens of related-file context. A sample-tune-sweep feedback loop lifted automation from 75% in the first four hours to 97%, with test intent and coverage preserved.

Do We Still Need Testers?
Sammy Pécce (LinkedIn)
Pécce relives a morning where testing nine prototype stories turned into design dialogue: her 'what if the user takes this path?' questions generated new user stories and Product conversations, plus retro shout-outs. With automation handling validation, she argues QA's durable value is a human engaging with the developer's thinking, challenging it warmly, and representing real users, even as developers orchestrate AI agents.

Hot Take 🔥

Why AI-Generated Test Suites Feel Weird
Wayne Roseberry (LinkedIn)
Roseberry diagnoses why every generative AI test suite feels weird: the suites are either trivially small or arbitrarily cherry-picked from an infinite space of combinations, because brute-force permutation expansion is infeasible and AI jumps straight to detailed cases, reproducing the cherry-picking. His fix: tools must mimic how human testers shift between specific cases and general patterns, which demands real testing knowledge injected as context. 'You are an expert tester' prompts don't do the job.

The VS Code Toggle That Turns Off All AI Features
Steven Boutcher (LinkedIn)
Steven Boutcher found the setting that disables every AI feature in VS Code and reports feeling much better: Cmd+Shift+P, Preferences: Open Settings (UI), then chat.disableAIFeatures. A two-line post, but the replies tell the real story: some engineers want an off switch so they can occasionally build something entirely from scratch. A small counterweight to a week of agent-first tooling news.

QA Folks, How Are You Feeling About Things These Days?
u/QualityAssurance (Reddit)
A r/QualityAssurance mood check draws gloomy replies: hiring freezes, waiting until we all get laid off, and testers who stopped learning new skills. Two counter-signals stand out: one CEO now calls QA non-negotiable in an AI-heavy stack, and QA interviews are adding live Claude prompting exercises.

The Sandwich Test: Flagship Models Roleplay Preferences, Old Models Refuse
Michael Bolton & James Bach (LinkedIn)
Bolton and Bach asked OpenAI models for their favorite sandwich. Flagship gpt-6-astra roleplayed a preference; elderly gpt3.5-16k honestly declined; Astra declined only under explicit non-anthropomorphic system instructions. Bolton argues the flagship's 'thinking' optimizes friendly tone over truth, and that buddy-style deception is a deliberate engagement feature. On truth and reliability, he concludes, the most capable model is now worse.

Is the Pendulum Ever Going Back?
u/QualityAssurance (Reddit)
Asked whether the pendulum will swing back, veteran tester u/DarrellGrainger answers with a typesetter's arc: computers killed his printing trade, he rebuilt into QA, and AI will do the same again. His takeaway, quoting Heraclitus: the river keeps flowing, so don't fight it. AI handles the grunt work; humans stay in the loop.

If something in this issue made you think differently about how your team approaches AI in testing, pass it along. The best conversations about AI and QA are happening in Slack channels and stand-ups, not just newsletters.

Have something worth featuring? Reply and send it my way, I read every link.

Thanks for reading,
Butch Mayhew