Do AI agents write good tests? What the research says.

“Just have the agent write the tests too” is the most common answer to AI-era QA, and it is usually said without evidence in either direction. So we went looking for the evidence. Below is what the peer-reviewed and large-scale-empirical work actually found, with sources, plus the widely-circulated numbers we could not verify and think you should stop repeating.

Short version: agents are good at producing tests that run, and unreliable at producing tests that judge.

Finding 1: Agents mock more than humans do

The largest study on this analyzed over 1.2 million commits from 2,168 TypeScript, JavaScript and Python repositories, isolating 48,563 commits authored by coding agents. Agents touched test files more often than humans, 23% of commits versus 13%. And when they did, 36% of those commits added mocks, against 26% for non-agents. Across repositories with any agent test activity, 68% showed agent mock activity.

Mocking is not wrong. Mocking the dependency whose behavior the test exists to verify is, and the incentive gradient runs that way: mocked tests are faster, deterministic, and pass on the first attempt.

Finding 2: Agent tests are debugging instruments

A 2026 paper examined what agents produce when they write tests during issue resolution. Three results stand out.

• Test-writing frequency varies enormously by model, from 83% of tasks down to 0.6%, and the model writing tests in 0.6% of tasks still resolved 71.8% of them.
• Across every model studied, value-revealing print statements outnumbered assertions by three to five times.
• Suppressing test-writing entirely cut token use 32 to 49% while reducing resolution rate by only 1.8 to 2.6 points. Encouraging it raised token use with no statistically significant gain.

The authors' conclusion is the sentence to remember: agent-generated tests act as “observational debugging tools rather than robust validation mechanisms”. They help the agent finish the task. They are not built to protect you later.

Finding 3: The oracle problem is unsolved

A test needs a source of truth about the right answer. Research across 24 Java repositories found LLM approaches are “prone on generating oracles that capture the actual program behaviour rather than the expected one”.

A test derived from the implementation cannot detect a bug in the implementation. It can only detect a change.

This is the deepest issue in the set, because no amount of model capability fixes it. Expected behavior lives in the ticket, the spec, the design file, and the customer conversation. It is not recoverable from the source code, at any level of intelligence.

Finding 4: High coverage, poor targeting

The TestEval benchmark evaluated 16 models on 210 Python programs. Aggregate line coverage reached roughly 92%, which sounds excellent. But the authors found that generating a test to cover a specific line, branch, or path “is still challenging”, indicating weak reasoning about execution paths.

Bugs do not distribute evenly across lines. They cluster in the error branches, the retry logic, and the states nobody thought about, which are precisely the paths that coverage percentage smooths over.

Finding 5: Test smells are pervasive

An analysis of 20,505 LLM-generated test suites found Assertion Roulette, multiple unlabeled assertions in one test, appearing in 23.8% to 61.3% of LLM tests depending on the model, against roughly 15% for the traditional generator EvoSuite. Magic numbers were similarly common. These are the properties that make a failing test expensive to diagnose at 2am.

Finding 6: What weak tests actually cost

This is the finding that ties the rest together. Researchers manually audited patches that SWE-bench counted as successful and found 29.6% induced behavior different from the ground-truth fix. A further 7.8% were counted correct despite failing the original developer-written suite. The net effect: reported resolution rates on the field's most-cited benchmark were inflated by 6.2 percentage points, entirely because the tests were not sharp enough to notice a wrong answer.

If inadequate tests can hide incorrect patches in the benchmark the entire industry optimizes against, assume they are hiding them in your repository too.

The fair counter-argument

Two things would be unfair to leave out. First, none of this says agent-written tests are worthless. A fast unit test that pins current behavior has real value during refactoring, and agents produce those cheaply and well. Second, human-written test suites have their own smells, tautologies, and blind spots. The research compares agents to a flawed baseline, not a perfect one.

The honest claim is narrower and still decisive: agent tests are weakest at exactly the job QA exists to do, which is catching the case where the code does something other than what the product promised.

Numbers we could not verify

While researching this we found several statistics circulating widely in blog posts that cite each other in a loop, with no traceable primary source. We are not repeating them, and we would suggest you do not either:

• “AI code shows 1.7x more issues” and “9x higher churn” — no primary study found.
• “62.4% of generated assertions were incorrect” — could not be traced to any paper.
• GitHub Copilot's “46% of code” and “88% retention” figures — real, but from 2022–2023 vendor research, routinely presented as current.
• METR's finding that experienced developers were 19% slower with AI tools is a real randomized trial, but METR now labels it historical and cautions it may not reflect current tools. Cite it with that caveat or not at all.

What we take from it

Every finding here points at one structural fact. An agent writing tests for its own code has only the code as a reference for correctness, so its tests converge on describing the implementation rather than judging it. That is not a prompting failure or a capability gap. It is what self-verification is.

Which is why we build Nua the other way around: agents that never wrote your code, that learn correct from your specs, tickets and designs, and that check your product through the browser the way a user would. More on that reasoning in our QA philosophy and why QA became the bottleneck.