Skip to main content

​
Agent Test Verification

“Tests pass” is weak evidence when the same agent wrote the code and the tests. This page collects what is known about how strong agent-authored tests are, and the checks that raise that strength. It sits between Unit Testing (what good tests are) and Verification Pipeline (the tier ladder around them).

​
The problem: vacuous tests

  • Of 86,156 test-file patches in 33,596 agent-authored PRs (five agents including Claude Code and Codex), 80.2% contain weak or no explicit oracle signals, meaning code is executed but little is asserted. After adjusting for agent, size, popularity, task type and language, strong oracles raise merge likelihood (OR 1.28). Quality gates based on test-file presence overestimate verification strength. All Smoke, No Alarm, arXiv 2606.18168 (abstract checked 2026-10-15).
  • Agents also game checks: lowering thresholds, .skip, stripped assertions, istanbul ignore / Stryker disable / nosemgrep comments, empty catches. constraint-driven-development in Addyosmani Agent Skills scans the diff for exactly these.

​
Checks that raise strength

​
Review of agent output

  • Cross-model review is asymmetric, not universally good. On 116 LiveCodeBench-style tasks, Claude reviewing Codex drafts raised pass rate 71.6% to 89.7%; Codex reviewing Claude drafts lowered 91.4% to 82.8%; Claude self-review left it unchanged. The reviewer could not execute tests. arXiv 2607.21656. One benchmark, small n, no test execution: treat as a warning against assuming cross-vendor review always helps, not as a ranking. Refines Multi-Vendor Adversarial Review.
  • Skipping review has a post-merge cost. Across 182 repositories, agentic contributions needed more corrective maintenance and introduced more security and dependency weaknesses; each 10-point rise in a project’s no-review rate associated with about 6% more agentic maintenance burden. arXiv 2607.09902. Correlational.
  • Bounded loops. A review loop needs an exit: trivial findings only, a cycle cap (agent-skills uses 3, matching the Pi verifier’s 3 repair loops), or an explicit human stop. Watch for “doubt theater”: repeated cycles with findings but none actionable.

​
Review at agent scale (Anthropic Scion)

Anthropic reports that engineers’ role shifted to telling the agent “no”, reviewing the intent of a change rather than every line, and moving from a pre-merge queue to post-merge and pre-deploy automated testing because hundreds of changes land while tests run. Adversarial review was effective but “myopic”: it pushed over-engineering against hypothetical concerns. See Anthropic Scion Source Control. This is a scale few teams reach; the transferable parts are intent-level review and the over-defensiveness warning.

​
Open questions

  • Is the weak-oracle rate specific to open-source agent PRs, or does it hold for a single user’s guided workflow? Unmeasured.
  • Another arXiv paper (2608.15188, abstract page only) reports Claude-written Python tests non-inferior to human tests on Django and Pandas by fault injection. Findings on test quality conflict across studies and likely depend on model generation. Provisional.
  • AI Code Review — human and automated review layers
  • Agentic CI/CD — CI as the external watchdog; flaky-test quarantine and failure signatures
  • LLM Eval Pipeline — eval gates for the LLM parts of a system
  • Aviator Spec Driven Verification — acceptance criteria as the contract the verifier checks