Agents don't review themselves
Transparency: we sell agent systems and diagnostics, and we use the systems we describe. The offer is at the end, labelled as such. The rest stands without it. Figures come from our own records; where a figure is a count after exclusions, from one run or from two runs, we say so.
The hidden tests were green. The specification had passed its check first time. And still, a review stage rejected the code.
This was a real run of our agent-driven software factory, on a small demo invoicing CLI of our own. A merge judge compared the change with the original request and found a bug that no test covered: the tool accepted nan and inf as amounts and produced invalid JSON output. The agent fixed it on the second iteration, the second review passed, the change was merged and the post-merge check came back green. The whole run cost about 0.53 USD.
That is a single case. It is not a rate and it is not a promise. But it shows what we want to say: an agent can deliver something that looks right, passes what it was asked to check, and is still wrong.
Why agents alone don't ship
An agent produces plausible output. The code compiles, the report reads well, the page looks good. What fails is usually what nobody asked it to check. If the agent that writes the work also approves it, approval is the likely outcome.
So what we sell is not an agent. It is a system around the agents, and a person at the end.
The layers, and what each one caught
1. Spec first, with a gate. Before any code, the request becomes given/when/then criteria, and a check confirms that every criterion has tests and that the new criteria fail on the current code. In our one complete run the spec gate passed on the first attempt. So far we have only seen it reject in unit tests, not in a real run.
2. Hidden tests the coding agent can't see. Whoever writes the tests is not whoever implements. There are two sets: one that returns "expected / got", and a sealed one that only says pass or fail. We have a test in which a cheating implementer memorises the answers of the first set, and the sealed set catches it. Again, a test, not a statistic.
3. Merge judge. A script blocks changes that touch the system's own configuration, secrets, migrations or more than 800 lines. Then a judge compares the change with the original request, not with the tests. This is the layer from the opening case: in that run the judge rejected the first of two reviews.
4. Claim verifier. When an agent drafts text, another agent checks every claim against a source. In one check of the drafts of our previous technical article and its posts, 18 of 61 claims had no support in the source repository. The rest: 40 verified, 2 derived by arithmetic and 1 contradicted (a title the draft gave as 66 characters long was 61). Several were rewritten before publication; the rest were kept because another source supported them.
5. Skeptic. In our agent research team, one agent has the sole job of trying to kill the idea. On one candidate the score went from 56 to 37 in the round where the skeptic reviewed it, together with a channel veto (one case). Of 26 opportunities researched in depth, 0 passed our bar of 70 out of 100 (the best reached 59). None of the 26 reached the 70-point bar.
6. Legal review. A legal review agent went through our own website and a draft of one of its pages. It flagged 10 numbered claims in the draft, 5 of them as legal risks to fix before publishing. They were, among others, of two kinds: absolute promises we cannot keep, and comparisons nobody could verify. They sounded good and they were ours.
7. The person. A senior engineer reviews and answers for every delivery. No earlier layer replaces that. One example from our own work: this studio's website has been made with agents. In human review, six homepage concepts were rejected before one was approved. It was noticed that the navigation bar changed from page to page ("it's annoying"). When the mobile view was checked, it turned out that below 860 px the links were hidden and there was no menu.
What it costs, using the research system as the example
This is the system where we have volume. Across its three instances it logged 15,019 tool calls; 12,639 remain after excluding entries that are not business research. On top of that, about 6,600 web searches and reads, about 300 subagent tasks (25 of them skeptic reviews) and 88 scored candidates, after exclusions.
For a full unattended round we have measured cost for only two: 18.12 USD and 40.8 minutes in one, 19.23 USD and 35.2 minutes in the other. So roughly 18–19 USD and 35–41 minutes, from two measured runs. Two measurements, not a reliable average.
Our reading, which is an opinion and not a measurement: when a whole round costs about 18–19 USD, the expensive part is not running the layers but finding out late that something wrong looked right.
Limits
- One complete real run of the software factory. There were at least 6 runs: 4 in our integration environment on 26 September (1 complete, 2 blocked before starting by a configuration fault, 1 stopped by an infrastructure fault), plus 2 local runs on 27 September that stopped at "ready" without merging. That is why we give no rates ("first time", "no reverts") and no average times: with n = 1 they would mislead.
- The automatic rollback has never fired outside tests. It is the feature we most want to demonstrate; it has not fired in any real run so far.
- We do not have the per-phase or total duration of that run, and the 0.53 USD is a sum of amounts that were already rounded.
- Rejection by the spec gate and by the sealed tests is only proven in unit tests.
- A research system that researches well does not mean it earns money. The cheap tests it opened have 0 EUR recorded.
What we will measure next: a batch of 10 to 20 real requests across 2 or 3 different products of our own, so we can give rates per layer (passed first time, rejections at each gate, rollbacks) and times as median and 90th percentile. We will trigger a real rollback and record it. We will publish the result even if it is bad.
What it means for a team adopting agents
- Separate whoever writes from whoever checks. The implementing agent must not see the acceptance tests.
- Check against the request, not only against the tests. Tests capture what you already thought of. Review looks for what you didn't.
- Cap spend and change size. A stuck agent should stop, not spend.
- Make failure reversible. Test after merging and roll back automatically.
- Verify text the way you verify code. In one check of the drafts of our previous article and its posts, 18 of 61 claims had no support in the source repository. It is one case, but enough not to publish unchecked.
- Put a person on the hook. Named, with time to review and the authority to say no.
This article also went through the verifier. The first draft had 4 figures that contradicted our own records (for example, 95 scored candidates when it was 88) and 9 claims with no support. We corrected them before publishing.
What we do for teams
Commercial section. Grawwww sells the services this article describes.
Agent systems, from 1,500 EUR; we start with a 300 EUR diagnostic.
The diagnostic (300 EUR + VAT, deducted if we go ahead) starts from a real case of yours: which tasks you would hand to agents, which gates are needed and where the human review would sit. You get a written report and a fixed quote before we start. The "from" price is the starting point; the fixed price goes in the proposal. If agents are not the right tool for your case, the report says so.
In the work that follows, a senior engineer reviews and answers for every delivery.
Send us your case: https://grawwww.xyz/en/teams/