Refinement Tax · executive read
Everyone quotes a figure for how much of agent spend goes on checking and fixing work rather than doing it. We tried to measure it on real hardware. The apparatus stopped itself after two hours and thirty-eight minutes, which is the most useful thing it could have done.
Two figures are shaping how organisations budget for AI agents this year. One holds that agentic workflows cost more than five times their nominal inference bill through 2028. The other puts about 60% of agent spend on refinement rather than first production, meaning the money goes into checking answers, repairing them and shuttling work between roles instead of into producing anything.
Sixty percent is a large number to be carrying around unmeasured. If it holds, the biggest line in an agent budget is quality control, and every efficiency argument that focuses on the model doing the work is aimed at the smaller half. So we built the smallest apparatus that could count it: a worker producing answers, a checker inspecting them, a router deciding what happens next, and a meter on the wall socket. Every model call gets tagged with exactly one of four phases, and every token and every joule belongs to the phase of the call that spent it.
Before any of that ran, the design was written down and frozen. Five hypotheses, each with a number attached and a stated condition under which it would be considered refuted. Ten gates, each with a threshold, each with a defined consequence. The point of writing it down first is that it removes the option of deciding what counts as success after seeing the data.
The campaign launched at 03:03 on 9 September. It passed 75 of 75 instrument checks by 03:42, calibrated the task difficulty and locked it at 04:25, ran a 32-task smoke stage, and shut itself down at 05:40 against one of its own gates. It never started production. The files that would hold production results do not exist, and none of the five hypotheses was tested.
The gate it hit asks whether the checker can tell good work from bad. That gate needs two groups to compare: tasks the model got right, and tasks it got wrong. At the difficulty the calibration stage had chosen, the smaller model got 0 of 16 right and the larger model got 16 of 16. Neither model produced both groups, so there was nothing to compare, and the campaign reported that rather than running eleven hours of compute to produce a number nobody could interpret.
The failure was in our own rule, and it is worth being precise about it
The frozen calibration rule picked the task difficulty whose average first-pass accuracy across the two models landed closest to 50%. It got exactly 50%. That average came from combining 0% with 100%, which is the single worst arrangement an average of 50% can conceal. A rule that instead required every model to sit away from both floor and ceiling could not have selected it. No threshold was moved before or after the run; what needs replacing is the statistic the rule reads.
This is the third campaign in forty-eight hours on this fleet to stop against the same problem, and all three were misled the same way. A pooled statistic looks reassuring precisely when it is uninformative. Averaging over the units you intend to compare destroys the one property you needed to check, which is whether each unit individually sits somewhere it can move.
The correction is cheap and it generalises well past this experiment. Anywhere a design plans to compare groups, calibrate each group separately and set the entry test on the weakest one. A group sitting at 0% or at 100% is a group that has left the study, and recording it as a data point at zero is how a floor gets mistaken for a finding.
After the stop, a diagnostic ran 120 tasks across three models and four difficulty settings to ask whether any setting would have worked. Both 8B models scored zero exact answers in all 80 of their tasks, at every difficulty. They parse the question, they engage with it, and they get the arithmetic wrong to the cent every single time. No difficulty setting on this task family separates two models on this hardware, which retires a design assumption that three campaigns have now paid for.
Joules are the novel half of this measurement and they came from a meter on the wall socket, which sees a whole machine and cannot see a process. So attributing energy to a phase of work requires that nothing else runs on that machine while the measurement is open. The campaign checked this rather than assuming it, by watching the request counter of the production service that shares the box.
Eight of the ten smoke measurements caught traffic that was not ours. The ceiling written down in advance was two in ten. One measurement window absorbed 54 requests belonging to someone else.
The arithmetic behind that is worth understanding, because it is counter-intuitive and it will recur. One outside request anywhere inside a measurement window spoils the whole window. Windows have to last minutes, because a meter sampling every six seconds needs hundreds of readings to integrate a curve. Outside traffic arriving at under two requests a minute therefore touches most windows. Longer windows buy better energy resolution and worse isolation, and there is no setting where both improve together on a shared machine.
The same interference showed up in speed. The larger model produced 35.5 output tokens a second during the clean hour and 15.1 during the window that absorbed 37 outside requests, on the same task at the same phase of work. That has a budget consequence beyond this campaign: the seven-hour estimate that got the run approved was built on throughput measured while the machine was quiet, and at the rates observed the same work lands between nine and eleven and a half hours against a limit of eleven.
The practical instruction is short. Per-task energy attribution needs a machine nobody else is using, and that is a scheduling decision about the fleet rather than something a single project can assert about itself. Take the host out of service for the duration, use a machine that is not in service at all, or measure tokens and say plainly that energy was not attempted.
The instrument check caught three defects before anything ran, and each one would have produced a confident, wrong, publishable number.
The grader was forgiving a one-cent error on a task made entirely of cent arithmetic, so rounding drift would have scored as success. The claim that the measurement was repeatable turned out to be false on the real prompts, traced to prompt-cache state and to a model being partially pushed onto the CPU when memory ran short, which changes the arithmetic path and therefore the answer. And the token budget was so tight that the largest model burned all of it and returned nothing parseable on six of six calls, which would have looked exactly like a model that could not do the task.
Then the gates caught the fourth, which was the design's own calibration rule. The whole apparatus cost two hours and thirty-eight minutes and it declined to spend the next nine.
The case for gates is usually made in the abstract. Here it is arithmetic. A campaign that runs to completion and then discovers its comparison was between a floor and a ceiling has spent the full budget and produced a number that reads as evidence. This one spent 24% of the budget and produced a design correction, a retired assumption about the fleet, and a hard constraint on where energy work can run. All three carry into the next campaign; a wrong number would have carried into a report.
| Reading | Value | What it settles |
|---|---|---|
| Coordination cost | 25 tokens | Per routing decision, 1.8% of all output. Wherever the refinement tax lives, it is not in the handoff between roles. |
| Checking cost when nothing is wrong | 40% | Share of the larger model's output tokens spent on verification, on 16 tasks it got right first time. Descriptive, from smoke only. |
| The 8B floor | 0 of 80 | Exact answers from two 8B models across all four difficulties. Cent-accurate multi-step arithmetic is out of reach for them here. |
| Meter accuracy | sound | Idle baseline 58.0 W, a known-quiet 300 s window attributing 171 J against a 1,500 J budget, a busy interval at 22,842 J and 155 W. Refusals return error codes and never a zero. |
Table 1. The energy instrument is fine. The machine is not quiet. Those are separate problems and only the second one blocks this work.
The checking figure is the one to watch. A model that needed no repair at all still paid four tokens of inspection for every ten it spent producing an answer, because the check runs on every task whether or not anything is wrong. That is a fixed toll rather than a cost of failure, and if it holds up on a model sitting somewhere between floor and ceiling, it changes how an agent workflow should be priced. Sixteen tasks at one difficulty on one model is not enough to say so.
A successor is worth running and the data says exactly where to point it. One model, qwen3.5, at difficulty level 2, where it scores 80% first time. At 50 tasks that gives roughly 40 correct and 10 incorrect answers, both groups comfortably large enough for the checker gate, with a 1.9% chance of falling short. Level 3 looks similar on the surface at 90% and fails that same test 43% of the time, so the choice is not close.
Three of the five hypotheses survive on one model. The magnitude question, whether refinement is more than 40% of the bill, becomes a token-only measurement and is reported as such. The within-model question, whether the workflow concentrates its effort on defective work, is the one worth the compute. The horizon question, whether refinement grows faster than the work it refines as tasks get longer, holds up on the task-level data. The between-model comparison is dropped rather than run over two floors and a ceiling.
The stop cost two hours and thirty-eight minutes. It arrived before eleven hours of compute rather than after, and it came from a rule written down before anyone had seen a result.
How to read this. Nothing here was tuned after seeing an outcome. The design, its five hypotheses, its ten gates and every threshold were frozen before the first task ran, and none was moved before or after. Where a number in the campaign's own write-up disagreed with the raw logs, the log's value is used and the disagreement is recorded in the accompanying technical paper. That paper, A Pilot That Stopped Before It Measured Anything, carries the full method, the threats to validity, and three corrections to figures published earlier in the day. This campaign appears with the others that stopped at a gate on The studies the gates stopped.