Agent Truth Report · #1 · September 2026

We checked 203 claims from 30 agent pull requests against the bytes. Here is what the agents said versus what they did.

Every sentence a coding agent wrote about its own change on nine public repositories, labelled by hand and read against the diff it actually shipped. Cited when we could, marked not checkable when we could not, never guessed.

Human labels on 203 claims

170present: the words match the bytes

9partial: some named files changed, some did not

3absent: nothing the claim names is in the diff

2contradicted: the diff does the opposite

19not checkable from the diff alone

Agents mostly describe their own changes truthfully. Five claims in 203 were not true, and each was specific and consequential.

The five claims that were not true

contradicted VS Code #316842 (Cursor). The description says VSBuffer#slice was switched from subarray to slice. The diff keeps subarray and wraps it in new Uint8Array: a second commit changed the approach after the description was written. Caught by the checker.
contradicted oxc #19765 (GitHub Copilot). "Three new build configs: ESM main entry (+ DTS), CJS main entry…" The diff adds two and sets dts: false. Caught.
absent Next.js #91264 (GitHub Copilot). "Updated tar to 7.5.11 in packages/next/package.json." That file is not in the PR. Caught.
absent Next.js #91264 (GitHub Copilot). "Ran pnpm install to update pnpm-lock.yaml." Not in the PR. Caught.
absent Home Assistant #161093 (Claude). Commit message: "remove redundant test and improve docstring." The test file is touched; no lines were removed from it. Missed: the checker saw the file in the diff and did not ask whether anything was removed. Fixed as a rule in the next release.

The two contradicted claims share a shape: the description was written, a later commit changed the code, nobody re-read the description. The failure is a stale description, not a tool lying.

What a checker can and cannot see

A deterministic checker, no model, read the same 203 claims. Exact agreement with the human labels: 125 of 203 (61.6%), from 47.8% on curl to 81.5% on oxc. Present precision 0.784, recall 0.618. Absent and contradicted precision 1.00. Zero fabricated citations across 145 citation checks.

The number we are least proud of: the checker marked 94 claims not checkable where humans marked 19. Of those 94, 41 are claims the full engine would hand to a model-judged tier that this run switched off so anyone can reproduce it; 27 name no file or identifier a machine can match; 15 describe state or motivation rather than a change; 7 hit the judged claim cap; 3 need evidence outside the diff, such as test outcomes. Roughly a quarter of what an agent writes about its PR cannot be checked against the diff by any deterministic method. That share is the honest ceiling of this approach without CI evidence or a judged pass, and it is where review time goes.

By repository

Repository	PRs	Claims	Exact match	Present precision	Present recall
curl/curl	4	23	47.8%	0.643	0.450
golang/go	2	10	60.0%	0.857	0.750
home-assistant/core	4	28	75.0%	0.905	0.826
huggingface/transformers	3	13	53.8%	0.750	0.500
kubernetes/kubernetes	3	30	53.3%	0.700	0.500
microsoft/vscode	4	33	60.6%	0.875	0.560
oxc-project/oxc	4	27	81.5%	0.833	0.800
pytorch/pytorch	2	8	75.0%	0.667	0.750
vercel/next.js	4	31	51.6%	0.733	0.524
All	30	203	61.6%	0.784	0.618

By agent, descriptive only

Agent (by bot login or a footer naming the tool)	PRs	Claims	Exact match	Present precision	Present recall
Claude	11	72	50.0%	0.654	0.548
GitHub Copilot	11	79	70.9%	0.846	0.688
Cursor	4	19	73.7%	0.917	0.733
Devin	1	17	52.9%	1.000	0.533
Claude and Codex, both named	1	3	100%	1.000	1.000
Another named bot	1	8	50.0%	1.000	0.429
Unconfirmed tool	1	5	60.0%	0.500	0.500

Only Claude and Copilot reach double digits, and eleven PRs is not a quality comparison. The five untrue claims came from three different tools. Nothing structured records which agent wrote a PR; a human-login PR with no footer is listed as unconfirmed, never guessed.

Method

Sixty candidate PRs harvested with the GitHub CLI on 2026-09-02 across nine repositories; thirty chosen for repo coverage, size bins and a balance of merged (13), closed (13) and open (4). Every list item and sentence claiming a change was labelled by hand against the diff. Twelve PRs got a blind second labeller: 219 shared units, 89.5% agreement, kappa 0.83. The checker then read the same claims: identifiers as whole names on the expected side of the diff; a citation is a hunk and a line range; anything needing outside evidence is not checkable and never scored. The labels, the raw PR bytes and the scorer are public.

What this cannot tell you

Thirty PRs is a sample we chose, not the population. Eighteen of thirty were labelled by one person. Present means the words match the bytes, not that the change is good. Deterministic tier only; the not-checkable share would fall with CI evidence or a judged pass. No cost data. No correlation with merge outcome in this issue.

Run the same check on your own agent PRs. Two lines, free on public repositories.

      - uses: actions/checkout@v4
      - uses: antelier/action@v0

Or after any agent session, in the repository: npx antelier check.