By Gustavo Paixão
Checking AI's work with an agent that didn't do it
Before I tick a task, a second agent checks it against the real code. The idea came from a paper on catching deceptive AI that I can't run. What I took from it, what it found and what it costs.
The problem
When an AI agent finishes a task, it tells me what it did. Most of the time that's accurate. Sometimes it isn't: a test that doesn't cover what it claims to cover, a case nobody tried, a plan that no longer matches the code. On Ronne AI Marketplace, 143 of the 1,522 claims the implementing agent made about its own work turned out to be only partly true, or not true at all.
So before I tick a task, another agent checks it. It doesn't write the code, doesn't fix it, and doesn't take anyone's word for it: it reads the files and runs the commands itself. I call it the state witness, after the paper the idea came from. Its record goes in a WITNESS.md next to the feature's spec and plan, in the same commit as the task, and a check in the pre-commit hook and in CI refuses a ticked task unless its latest witness pass confirmed every claim.
I write the spec and mark which tasks are risky; the implementing agent starts the witness and pastes its record into the file.
Where the idea came from
I first heard about StateWitness in an Instagram Reel by Vinicius Pasquantonio (@pasquadev). It's a research paper (Chen et al., 2026) about catching deception in reasoning models. A second model reads the activations of the model being audited (its hidden states) and answers questions such as "does this answer hide something?". Across seven deception benchmarks it reached a mean AUROCArea Under the Receiver Operating Characteristic CurveIn AI and machine learning, a metric of how well a classification model tells two classes apart, such as:Buggy vs. clean commitsFlaky vs. reliable testsMalicious vs. safe packagesVulnerable vs. safe codeDeceptive vs. honest AI answers, as in this paper of 0.916, against 0.821 for the best judge that only reads the text.
I can't run that. The method only works on a model whose activations you can read:
Open-weight models
- Examples
- gpt-oss-20b and Gemma, the two in the paper
- What you get
- The weights. You run the model yourself, so you can read its activations.
- A full witness
- Possible, with a decoder trained for that exact model and GPUs: the authors report about 190 A100 GPU-hours of training.
Commercial models
- Examples
- GPT, Gemini, and Claude, the model behind Claude Code
- What you get
- Text, through an app or an API.
- A full witness
- Not from outside. Only the company that runs the model sees its activations.
The same companies make both kinds: gpt-oss and GPT come from OpenAI, Gemma and Gemini from Google. With Claude Code, the activation reader is out of reach, and what's left are the ideas that work on text.
The paper also answers a different question. It asks whether a model is being deceptive, and I need to know whether a task is done.
What I could use were three findings about checking someone else's work, plus one rule of my own.
What I took from it
| Finding | In the paper | In the witness |
|---|---|---|
| The story anchors the checker | A text judge did worse when it also read the model's reasoning: 0.789 against 0.809 on gpt-oss-20b | It starts blind: the task's Done when (the plan's line that says what counts as finished), the spec lines and where to look, never the implementer's notes |
| Two different checkers miss less than one | Combining the activation reader with the text judge raised recall at a 1% false-alarm rate from 0.430 to 0.562 | Risky tasks (sign-in, roles, migrations, deleting data, anything that runs as root) get a second pass, told only to break the claims |
| A checker can sound sure on weak evidence | The authors say so in their limitations | Every claim needs evidence: the command and what it showed, or a file and line. Every pass names the commit it checked |
The first finding is the weakest of the three. On the paper's other model, Gemma, the judge did slightly better with the reasoning.
| Final answer only | Answer and reasoning | |
|---|---|---|
| gpt-oss-20b | 0.809 | 0.789 |
| Gemma 4 26B | 0.826 | 0.829 |
AUROC from the paper's main results: 1.0 catches every deceptive answer with no false alarm, 0.5 is chance. Bars start at zero.
I kept the blind start anyway. It costs nothing, and my own numbers below point the same way.
The rule that's mine, not the paper's: after its verdicts, the witness reads the implementer's notes and marks which claims they made. A failed claim the notes made is one the implementer got wrong. Those are the 143 at the top.
How a pass works
A fresh agent gets the task's Done when, the spec lines it covers, where to look and the commit. It turns them into separate claims, one per row, and checks each one by reading the code and running the tests, builds and small probes itself. Green tests alone don't count: it checks that a test covers the claim and, where it can, breaks the code in a scratch copy to see the test fail.
Each verdict is one of four: confirmed, partly, not met, or can't check here (it needs Windows, GitHub or a release). These rows come from a real adversarial pass on a change to the dependency resolver:
| # | Claim | In the notes? | Verdict | Evidence |
|---|---|---|---|---|
| 30 | The core suite passes, with the many-askers test | no | confirmed | 350 passed |
| 31 | The change passes pnpm lint | no | not met | noAssignInExpressions at resolve.ts:224 |
| 32 | The budget counts range checks, so many askers can't run long | no | confirmed | N=100/300/500 → 454 / 1159 / 4391 ms |
| 33 | The new test fails when checks count +1 per version | no | confirmed | → 11,543 ms, fails |
The tests were green. The lint wasn't.
What it found
Every figure in this note comes from the records as they stood in the repository on October 11 at 22:21 Eastern time (commit 149e14a (opens in a new tab)), eight days after the witness started, and from the Claude Code transcripts up to that commit. The repository is public, so the record figures can be recounted.
In those eight days the witness ran 408 passes over 3,490 claims: 3,122 confirmed, 268 partly, 89 not met and 8 it couldn't check here (3 early rows carry a verdict the format no longer allows). The 1,522 claims at the top are the ones among the 3,490 that the implementer's notes had made. The 408 passes came from 251 witness runs: a re-check usually goes back to the agent that did the pass, with the fixes, rather than to a new one.
Partly is the common failure, and it covers a wide range. In the record: a test that exists but doesn't assert the field the claim is about; a race test that's flaky on SQLite, failing once in twelve runs; a documentation section numbered "fourth" when it was the third. Some of those are the implementer's mistakes and some are the witness being strict, and the record doesn't say which.
The first version gave the witness the implementer's notes along with the task. After reading its first 55 passes, on 42 tasks, I changed the rule so it starts blind, and the share of tasks whose first pass left at least one claim unconfirmed went up sharply.
| Tasks | |
|---|---|
| Witness given the notes (first 42 tasks) | 14% (6 of 42) |
| Witness starting blind (next 125 tasks) | 58% (73 of 125) |
A before and after, not a controlled test: the later features, workspaces with roles and permissions, were also harder. Running the same tasks both ways is still on my list.
The second pass earned its place too. On 11 of the 38 risky tasks, the adversarial pass found problems after the blind pass had said everything was met. In general terms: information leaking between workspaces, a cache shared between administrators that should have been separate, and deadlocks on some database servers when two changes arrived at once. Each was fixed before its task was ticked.
It isn't quick. A task took 2.4 passes on average, and the worst one took 17.
What it costs
I use Claude on a Max plan, so the witness doesn't add to a bill. It uses up the plan: in those eight days, 23% of all the Claude usage on the project went to the witness. That's measured on the same transcripts and at the same prices: the witness's tokens against everything Claude Code recorded for the project. It also takes time. A witness run takes a median of 5 minutes, and the slowest tenth 29 minutes or more: 50 agent-hours in all. A task can't be committed until its pass is back. The fixes it sets off are the implementer's work and aren't counted here; I take them as the point of the exercise, not its cost.
If you pay per token, this is what the same runs would cost at Claude Opus 5.5's API prices:
| Kind of run | Runs | Cost | Per run |
|---|---|---|---|
| Blind | 173 | $186 | $1.08 |
| Adversarial | 39 | $107 | $2.75 |
| Re-check by a fresh agent | 39 | $26 | $0.65 |
| All | 251 | $319 |
A run's cost includes the re-checks the same agent went on to do. One risky task got its adversarial re-check from a fresh agent rather than the same one, which is why there are 39 adversarial runs for 38 risky tasks.
How I calculated it
These are the tokens Claude Code's transcripts recorded, with Claude Opus 5.5's prices from Anthropic's pricing page:
| Kind of run | Cache reads | Cache writes | Input | Output (estimated) |
|---|---|---|---|---|
| Blind | 266.7M | 16.86M | 0.01M | 2.44M |
| Adversarial | 176.8M | 8.97M | 0.004M | 1.35M |
| Re-check by a fresh agent | 36.5M | 2.32M | 0.001M | 0.33M |
| Price per million | $0.20 | $5 | $4 | $20 |
Multiply each column by its price and add them up. For the blind runs:
266.7M cache reads × $0.20 + 16.86M cache writes × $5 + 0.01M input × $4 + 2.44M output × $20 = $186.
- Cache reads are most of the tokens, because almost everything an agent reads comes from the cache. At $0.20 a million, they still cost less than the writes.
- Cache writes are priced for the 5-minute cache, $5 a million rather than the 1-hour cache's $8: the transcripts record which lifetime each write bought, and none of the witness's used the longer one.
- Output is an estimate. The transcripts don't record it for the checking agents, so I used the tokens-per-character rate of my main sessions, which do.
- The transcripts run from October 4, the witness's first day, to the commit above.
Was it worth it
Across all 38 risky tasks, the adversarial runs cost about $107 at API prices and found 51 claims that didn't hold and 104 that held only partly, the leaks between workspaces among them. For software whose job includes keeping one team's work away from another's, a single leak in production would cost far more than that.
What I can't measure yet is what the witness misses, or how often it's wrong. My process asks for a test once a milestone: plant a bug the task should catch and see whether a fresh pass finds it. I haven't run it yet. That's next.
What I'd change
- A second checker that's really different. The paper's gain came from two monitors reading different signals. Mine reads the same signal twice: same model, same instructions, a different brief. I'll try another model for the adversarial pass, and mutation testing, which changes the code slightly and checks that the tests notice.
- Adversarial passes beyond the tasks I mark risky. It found problems on 11 of 38 risky tasks. I don't know the rate on the rest, and a random sample would tell me.
- A full re-check before the tick. Today a re-check can cover only the claims that failed, so earlier claims aren't run again on the final commit.
- A record of the witness's own mistakes. Nothing in a pass says whether a not met turned out to be the witness's error, so I can't give its false-alarm rate. One more column would.
- Severity on each finding. A wording slip and a data leak count the same in the record today.
The witness's instructions are in the repository, with the reasons behind each rule.
Read the witness's instructions
If you check your agents' work a different way, I'd like to compare notes.