We should say this first: we use Claude Code every day, and this is not a hit piece. It is one of the best coding tools ever built, and Anthropic's own documentation is unusually honest about where it falls down.
That honesty is the problem. The guidance for getting good tests out of Claude Code is real, specific, and almost nobody reads it. What most teams get instead is the default behavior, and the default behavior has four failure modes that are now well documented in the research literature.
1. It writes tests to debug, not to guard
When an agent writes a test mid-task, it is usually trying to answer a question for itself: what is this function actually returning right now? That is a legitimate and useful thing to do. It is just not the same artifact as a regression test.
A 2026 study of agent test-writing across models measured exactly this. Value-revealing print statements outnumbered assertions by three to five times. The researchers concluded that agent tests function as “observational debugging tools rather than robust validation mechanisms.” The same paper found test-writing rates ranging from 83% of tasks in one frontier model down to 0.6% in another, with no meaningful difference in how often the underlying problem got solved.
Debugging scaffolding that survives into your test suite looks like coverage. It gates nothing.
2. It mocks the thing it was supposed to test
Anthropic's best practices doc contains a small example that gives the whole game away. It contrasts a weak prompt, “add tests for foo.py,” with a strong one: “write a test for foo.py covering the edge case where the user is logged out. avoid mocks.”
They wrote that instruction because the default pulls the other way. Researchers analyzing 48,563 agent commits across 2,168 repositories found agents introduce mocks in 36% of test-touching commits, versus 26% for humans. Mocking is the path of least resistance for an agent, because a mocked test is fast, deterministic, and passes on the first run.
A heavily mocked test asserts that your code called the functions you told it to call. Your users do not call functions. They click things.
3. It asserts what the code does, not what it should do
This is the subtle one, and the most dangerous. To write an assertion, you need an oracle: a source of truth about the correct answer. An agent looking at your implementation has one obvious oracle available, which is the implementation itself.
A study of LLM-generated oracles across 24 Java repositories put it plainly: these approaches are “prone on generating oracles that capture the actual program behaviour rather than the expected one”. If the function has an off-by-one error, you get a test asserting the off-by-one result. The bug is now load-bearing, protected by a green check, and the next developer who fixes it will break the build.
4. Coverage goes up. Targeting does not.
The TestEval benchmark ran 16 models against 210 programs and found high aggregate line coverage, around 92%, alongside a clear weakness: generating a test to cover one specific branch or path remains difficult. Models cover the easy majority of lines and miss the specific paths where bugs live.
Which means the number you are most likely to report to your team, coverage percentage, is the number least connected to whether the suite would catch a regression.
What Anthropic actually recommends
The docs are good. If you take nothing else from this post, take these, which come straight from Anthropic's guidance:
- Give it a check it can run.“Tests, a build, a screenshot to compare. It is the difference between a session you watch and one you walk away from.”
- Split the writer from the reviewer.Have one session write tests and a different one write the code, or review in a fresh context. Anthropic's reasoning: “a fresh context improves code review since Claude will not be biased toward code it just wrote.”
- Name the edge case and ban the mocks. Vague prompts get vague tests. Specify the scenario you actually fear.
- Do not let the worker grade the work. Use a separate model to try to refute a result rather than confirm it.
Anthropic even names the underlying failure mode, calling it the trust-then-verify gap: “Claude produces a plausible-looking implementation that does not handle edge cases.” Their prescription is blunt. If you cannot verify it, do not ship it.
Cursor's documentation lands in the same place, warning that AI code “might follow existing patterns, compile, and pass tests you wrote, but still miss edge cases”. Both vendors are telling you the same thing: the agent is not the gate.
The structural version of this problem
Every failure above reduces to one sentence. The author is grading its own work.
All four behaviors are what you would predict from a system optimizing for a green run, with the implementation as its only reference for correctness. Better prompting genuinely helps, and you should do it. But prompting cannot manufacture independence, and independence is the property that makes testing mean anything. That is why QA became a separate discipline decades before any of this, a story we told in our short history of software testing.
So use Claude Code to write your features. Use its own advice to get better unit tests out of it. Then put something in front of your users that did not write the code, does not know how it was implemented, and only knows what the product was promised to do. That is the part agentic coding has not solved for itself.
Nua is AI QA built for agentic coding. Test agents that use your product like a real user, judge it against your specs instead of your source, and never wrote the code they are checking.