AI for Software Testing: What QA Agents Can Take Over, and Where Human Testers Still Matter
AI for software testing means using models and agents to write, run, repair and triage tests, so that verification keeps pace with the volume of code AI assistants now produce. That volume is no longer marginal. Alphabet CEO Sundar Pichai said on the company's April 2025 earnings call that well over 30 percent of code checked in at Google involved people accepting AI-suggested solutions, and Google's 2025 DORA report found 90 percent of the nearly 5,000 technology professionals it surveyed using AI at work. The same DORA research found AI adoption still associated with less stable software delivery. The position taken here: AI can take over most of the labor of testing, but not the judgement of what correct behavior is. Organizations that let agents decide both what to test and what the right answer looks like will see coverage rise while defects keep reaching customers.

Key Takeaways
- Testing is now the constraint. AI raised the volume of code; Google's DORA research ties AI adoption to higher throughput in 2025 but still to lower delivery stability.
- AI is good at the labor of testing: generating unit tests, repairing broken UI scripts, producing test data, exploring applications and triaging flaky failures.
- AI-written tests have a structural weakness. Research shows they mainly capture what the code currently does, so a bug in the code becomes a bug in the test.
- Coverage is a weak proxy for quality. Measure mutation scores, escaped defects and change failure rate instead.
- The working model: humans own test intent and acceptance criteria, agents generate and maintain tests, and a second model or a human reviews AI-written code.
- Security, performance under real load, accessibility and domain-critical acceptance tests still need people with authority to say no.

Why Testing Became the Bottleneck
For most of the last decade, the limiting factor in software delivery was writing code. AI coding assistants changed that. Google's DORA program, which has surveyed software teams for over ten years, captured the shift in two consecutive reports.
The 2024 Accelerate State of DevOps report, from a DORA research program that has drawn on more than 39,000 professionals over a decade, estimated that every 25 percent increase in AI adoption was associated with a 1.5 percent reduction in delivery throughput and a 7.2 percent reduction in delivery stability. The result broke a pattern DORA had reported for a decade, in which speed and stability rose together.
The 2025 State of AI-assisted Software Development report, published by Google on September 23, 2025, found the throughput relationship had turned positive: teams using AI were shipping more. The stability relationship had not. Google's announcement states that AI adoption continues to have a negative relationship with software delivery stability. More code is moving faster, and more of it breaks.
Code-level analyses point the same way. GitClear, which sells code analytics software, analyzed 211 million changed lines from 2020 to 2024 for its 2025 AI code quality report. Moved code, a sign of refactoring, fell from about 25 percent of changed lines in 2021 to 9.5 percent in 2024, while copied and pasted code rose from about 8 percent to 12.3 percent. 2024 was the first year copy-paste exceeded moved code. Duplicated code is more code to test and more places for one fix to be missed, which is how AI output turns into the kind of debt described in how to measure technical debt.
The productivity picture behind this is contested, which is covered in depth in the evidence on AI coding assistant productivity. For testing leaders, the practical conclusion is simpler: whatever AI does to coding speed, it increases the volume of change that has to be verified.

What AI Does in Testing
Unit and integration test generation
Test generation is the most mature use, and the best public evidence comes from Meta. Its TestGen-LLM paper, published in February 2024, reported that in an evaluation on Instagram's Reels and Stories products, 75 percent of generated test cases built correctly, 57 percent passed reliably and 25 percent increased coverage. Across Meta's test-a-thons, engineers accepted 73 percent of the tool's recommendations for production.
The design matters as much as the numbers. TestGen-LLM did not hand engineers raw model output. It filtered candidates through a pipeline: does the test build, does it pass repeatedly, does it add coverage. Only survivors reached a human reviewer. The lesson for executives is that generation is cheap and filtering is where the value is created.
Mutation-guided test generation
Meta's next step addressed a weakness of coverage-driven generation. Its Automated Compliance Hardening system, described on Meta's engineering blog in February 2025 and in a paper at the FSE 2025 conference, first uses a model to inject realistic faults, called mutants, into the code, then generates tests designed to catch them. Applied to 10,795 Android Kotlin classes across seven Meta platforms, it produced 9,095 mutants and 571 privacy-hardening tests. Engineers accepted 73 percent of the tests and judged 36 percent privacy-relevant, according to the paper. A test written to catch a specific plausible bug is far more useful than one written to touch a line.
Self-healing UI tests
End-to-end UI tests break whenever a button moves or an element is renamed, and maintaining them has long consumed much of a test automation team's time. Self-healing tools detect the change and update the locator or step. Vendors make large claims: mabl says its agentic tester can eliminate up to 95 percent of test maintenance, and Tricentis said early adopters of its Agentic Test Automation, announced on June 24, 2025, reported up to 85 percent time savings in test creation. These are vendor figures without independent audit.
The open-source ecosystem moved in the same direction. Microsoft's Playwright framework added three built-in test agents in version 1.56 in October 2025: a planner that explores the application and writes a test plan, a generator that turns the plan into test code, and a healer that runs the suite and repairs failing tests.
Test data generation
Models can produce realistic synthetic records, edge cases and malformed inputs on demand: patient records with unusual medication combinations, orders with extreme quantities, addresses in rare formats. This is useful wherever production data cannot be copied into test environments for privacy reasons. The control is to confirm the synthetic data covers the distributions that matter and contains no real personal data.
Exploratory testing by computer-use agents
Agents that operate a browser or desktop like a person can now explore an application without a script, a pattern covered in the design guide to computer-use agents. Research is early. Columbia researchers' WebProber prototype, presented at a NeurIPS 2025 workshop, explored 120 academic personal websites and found 29 usability issues, many missed by traditional tools. The authors of WebTestPilot, a 2026 paper on agentic web testing, note the central problem: when an agent flags an inconsistency, it is unclear whether the agent hallucinated or the application has a real bug. Agent exploration produces leads, not verdicts.
Flaky test triage
A flaky test passes and fails on the same code. Google's testing team reported in 2016 that about 1.5 percent of its test runs produced a flaky result, that almost 16 percent of its tests showed some flakiness, and that about 84 percent of pass-to-fail transitions in its post-submit testing involved a flaky test. The figures are a decade old but illustrate the scale. AI is well suited to clustering failures, reading logs and separating infrastructure noise from real regressions, which returns engineering time to real defects.
Review of AI-written code
Agents now write whole changes, not snippets. GitHub made its Copilot coding agent generally available to paid subscribers on September 25, 2025, according to GitHub's changelog; it works asynchronously and requests a human review when it opens a pull request. That makes review the control point. A second model reviewing the first model's change, before a human does, is now a common pattern, and it is only as good as the criteria it reviews against.
The Evidence on Quality When AI Writes More Code
The evidence is mixed, and its sources have different incentives. The table separates independent research from vendor and vendor-adjacent work.
| Source and date | Finding | Who produced it |
|---|---|---|
| DORA 2024 report (Google) | 25 percent more AI adoption associated with 1.5 percent lower throughput and 7.2 percent lower stability | Google research program; DORA research has drawn on more than 39,000 professionals over a decade |
| DORA 2025 report, Sept 2025 | Throughput now positive; stability still negative | Google research program, nearly 5,000 respondents |
| METR, July 2025 | Experienced open-source developers took 19 percent longer with AI, while believing they were about 20 percent faster | Independent nonprofit, randomized trial, 16 developers, 246 tasks |
| Uplevel, 2024 | 41 percent higher bug rate in the group with Copilot access, no throughput gain | Engineering analytics vendor, 785 developers |
| GitClear, 2025 | Refactoring down, copy-paste up across 211 million changed lines | Code analytics vendor |
| Veracode, July 2025 | Code from over 100 models chose an insecure approach in 45 percent of 80 coding tasks; Java failed 72 percent | Application security vendor |
| Stack Overflow survey, July 2025 | 46 percent of developers distrust AI output accuracy; 66 percent cite solutions that are "almost right" | Developer community survey |
Three cautions apply. METR's sample was small and its authors warn against generalizing. Uplevel measured access to Copilot, not usage. Veracode and GitClear sell products that address the problems they measure. Taken together, though, the direction is consistent: AI raises output, and verification has not kept up.
Where AI Testing Fails
Tests that assert what the code does, not what it should do
This is the most important failure, and it is structural. A test has two parts: the steps that exercise the code and the oracle, the assertion that defines a correct result. When a model writes a test by reading the code, it tends to assert whatever the code currently returns.
Researchers Michael Konstantinou, Renzo Degiovanni and Mike Papadakis studied this across 24 Java repositories in a paper first posted in October 2024 and revised in October 2026. The revised version concludes that LLM-based test generation approaches "mainly capture the actual program behavior making bug detection difficult." A January 2026 paper by Adam Bodicoat, Gunel Jahangirova and Valerio Terragni found that existing techniques largely generate regression oracles that encode implemented behavior, and that prompting strategy affects oracle quality more than the choice of model. If the code computes the wrong discount, the generated test confirms the wrong discount.
Coverage that rises while escaped defects do not fall
AI makes high coverage cheap, which makes coverage a dangerous target. Laura Inozemtseva and Reid Holmes, in an ICSE 2014 paper that tested 31,000 suites across five systems, found only a low to moderate correlation between coverage and fault detection once suite size was controlled for, and recommended against using coverage as a quality target. A dashboard showing coverage climbing from 60 to 90 percent says little about whether customers will see fewer defects.
Missing domain knowledge
A model can test that an insurance quote calculates; it cannot know that a regulator capped a fee, that a hospital formulary forbids a drug combination, or that a retailer promised price matching in one region only. Acceptance criteria encode business knowledge that lives in people, contracts and regulations, not in the codebase.
Security and performance
Functional tests do not find most vulnerabilities, and Veracode's finding that AI chose insecure methods in 45 percent of its tasks shows the code under test is part of the risk. Performance testing depends on realistic load, data volumes and infrastructure, and a model has no view of production traffic.
Healing over a real bug
A self-healing test that adapts to a changed screen can also adapt to a broken one. If a checkout button disappears because of a defect and the healer finds another path, the suite passes and the bug ships. Every automated repair needs a log and a rule for which repairs require a human to accept.
How to Structure the Work
Humans own test intent and acceptance criteria
The scarce input is the definition of correct. Product owners, domain experts and test leads write acceptance criteria, examples and invariants in plain language before code is generated. Agents then write tests from the specification, not from the implementation. This one change addresses the oracle problem more directly than any tool choice.
Agents generate and maintain
With intent fixed, agents take the volume: unit tests, test data, locator repairs, regression suites, failure triage. Every agent-generated test passes the same filters Meta used: it builds, it passes repeatedly, and it adds something, ideally by catching a seeded mutant.
A second model or a human reviews AI-written code
AI-written changes get a review against stated criteria by a different model or a person, and high-risk areas, such as authentication, payments, clinical logic and safety controls, always get a person. Testing agents themselves need evaluation, a practice covered in the guide to agent evaluation and testing.
Measure what customers feel
Replace coverage targets with three measures: mutation score, which shows whether tests catch injected faults; escaped defects, found in production rather than before release; and change failure rate, the DORA measure of how often deployments cause incidents. If AI testing works, all three improve. If only coverage improves, it is not working.
The Decision by Test Type
| Test type | Decision | Why | Control |
|---|---|---|---|
| Unit tests for well-specified functions | Automate fully | High volume, cheap to verify, failure is local | Build, repeat and mutation filters |
| Regression suites for stable features | Automate fully | Encoding current behavior is the purpose | Periodic pruning of redundant tests |
| UI locator repair and flaky triage | Automate fully | Mechanical maintenance | Repair log; human sign-off when a flow changes |
| Test data generation | Automate with review | Coverage of edge cases matters | Check distributions; no real personal data |
| Integration and API contract tests | Automate with review | Contracts encode decisions between teams | Owner approves contract changes |
| Tests generated for new features | Automate with review | Oracle problem is acute in new code | Generate from acceptance criteria, not code |
| Exploratory testing | Automate with review | Agents find leads, with false positives | Human confirms each reported defect |
| Security testing | Keep human-led | Adversarial judgement and liability | AI scanning assists; specialists decide |
| Performance and resilience | Keep human-led | Needs production-like load and context | Engineers design scenarios; AI analyzes results |
| Acceptance of domain-critical rules | Keep human | Business, legal and safety knowledge | Named owner signs off |
| Accessibility and usability | Keep human-led | Real user experience is the test | Automated checks plus human sessions |
The recommendation: automate the maintenance and volume work first, because it frees the most time at the lowest risk, and route all new-feature test generation through human-written acceptance criteria.
Examples Across Industries
- Healthcare. A hospital system integrating clinical applications can let agents generate interface tests for message formats while clinicians define the rules for drug interactions and alerts, which no model should infer from code.
- Retail. A retailer can hand checkout and catalog regression suites to self-healing agents, given how often storefront screens change, while merchandising owns the acceptance tests for promotions and pricing rules.
- Manufacturing. A manufacturer modernizing plant software can use AI to pin current behavior with characterization tests before change, the same approach described in AI for legacy code migration, with engineers owning safety interlocks.
- Public sector. A benefits agency can use agents for test data and regression, while eligibility rules, which come from statute, are specified and accepted by policy staff.
- Software and SaaS. A product company can let agents maintain most of its unit and UI tests and point its human testers at exploratory, security and performance work.
The Position, and Its Tradeoff
The position: let agents own the labor of testing, but keep the definition of correct behavior with named humans, and judge the program by escaped defects rather than by coverage.
The tradeoff is up-front effort. Writing acceptance criteria before generation slows the start of each feature, and product teams often resist it, because a model can produce a passing suite in minutes without it. The organizations that accept the slower start get tests that catch bugs. The ones that skip it get large green dashboards that verify the code is consistent with itself.
Frequently Asked Questions
Can AI replace QA testers?
AI can replace much of the repetitive work: writing routine unit tests, repairing broken scripts, producing test data and triaging failures. It cannot replace the judgement of what correct behavior is, adversarial security testing or domain acceptance. The QA role shifts toward defining intent, reviewing agent output and testing what agents cannot.
Are AI-generated tests reliable?
They are reliable at exercising code and unreliable at judging it. Meta reported that 57 percent of its generated tests passed reliably in one evaluation and filtered out the rest before review. Research shows generated assertions tend to encode current behavior, including bugs, so tests should be generated from specifications and checked against seeded faults.
What should be automated first with AI testing tools?
Maintenance and volume work: UI locator repair, flaky test triage, regression suites for stable features and test data. These free the most time at the lowest risk. New-feature tests should come next, generated from human-written acceptance criteria.
How should AI testing be measured?
By mutation score, escaped defects and change failure rate, not coverage. Coverage is easy for AI to raise and weakly related to fault detection, as a widely cited 2014 study of 31,000 test suites found.
The Bottom Line
AI has made code cheap, and testing is where that cheapness is either caught or shipped. Agents can now carry most of the labor of verification, and organizations that refuse them will fall behind on maintenance alone. The judgement of what the software is supposed to do still belongs to people. Executives who fund agents for test volume and fund humans for test intent will get faster delivery with fewer escapes; those who fund only the agents will get more tests and the same incidents.