Refinement Tax · pilot report

A Pilot That Stopped Before It Measured Anything

The campaign asked what share of an agent workflow's tokens and joules goes to verification, repair and coordination. It ran two hours and thirty-eight minutes, failed its own smoke gate, and attempted zero production tasks. Two things it learned on the way out matter more than the number it was built to produce.

Dinand Tinholt · 9 September 2026
Pre-registered 2026-09-09 before any production task · every figure below re-derived from the raw ledgers on the host

1The question, and why anyone should care

Two numbers are steering procurement decisions right now and neither has been measured in public. One says agentic workflows cost more than five times their nominal inference through 2028. The other says about 60% of agentic spend goes on refinement rather than first production. If the second figure is anywhere near right, then the tokens and joules that go into checking work, fixing it and routing it between roles are the dominant line item in every agent budget being written this year, and they are being estimated rather than counted.

This pilot was built to count them on one workload. A worker produces an answer, a checker inspects it and writes a defect report, a router reads only the checker's verdict and decides where the task goes, and the worker revises. Every model call carries exactly one phase tag out of first_pass, verification, coordination and repair, so the four partition the spend with no residual. The refinement share is the last three divided by all four, computed separately in tokens and in joules.

The router is what makes coordination measurable here. It sees the checker's verdict and nothing else, with the answer and the underlying task both hidden from it. A coordination figure defined as leftover tokens would be unfalsifiable, and this one carries no task content and does no checking, so its cost is the price of running more than one role.

2Design

The task battery is expense-claim reconciliation. N line items arrive in mixed currencies with per-category caps and one tax rate; convert, cap, subtotal, tax, total, and name the lines the cap bound. Ground truth is computed in Python with decimal and round-half-up, so an answer is right to the cent or it is wrong. There is no LLM judge anywhere in the grading path, because putting one there would insert a second capability-dependent instrument into a capability-ladder experiment and make any ladder effect unattributable between the workers and the judge.

Horizon N runs over {2, 4, 8, 16}, three octaves, so the horizon regression has range. Fifty production tasks were weighted toward the cheap end at 15/15/12/8. Difficulty runs on a four-level ladder that changes only knobs: currency count, decimal places on the exchange rates, the target rate at which caps bind, and the tax rate. The rules of the task are identical at every level, so moving the level cannot change what counts as correct.

Three model rungs survived pre-launch probing. qwen3:8b and llama3.1:8b serve through ollama; qwen3.5 (a 35B mixture-of-experts) is the standing vLLM tenant on the same box, read over HTTP and never stopped or reconfigured. Four candidates were excluded with a measured reason each, including gemma3:27b on cost alone: at 11.5 tokens/s and 1,127 mean completion tokens it needed about 6.8 hours for that rung by itself, which would have put the run past its cap.

Energy, and what a whole-machine meter can and cannot do

Joules come from a Tapo P316M strip poller that samples outlet 3 every six seconds and reports whole watts. It sees a machine, with no visibility into a process or a container or a model. So per-phase energy rests on two things that the design checks rather than assumes: exclusive occupancy of the host, and a measured idle baseline taken over a quiet 300-second window immediately before production and again afterwards.

Six-second cadence drives the execution shape. A single model call lasting a few seconds cannot be integrated from two samples, so the pilot runs phase-batched: every call of one (model, round, phase) executes as one contiguous block of tens of calls lasting minutes, bracketed by 20-second idle guards. Task dependencies survive intact, since verification of task i still consumes the first-pass output of task i; only the scheduling changes. Joules per phase per model then get integrated directly over hundreds of samples. Per-task joules are apportioned within a block by call duration, labelled derived rather than measured, and no hypothesis rests on them.

A block is refused, with an error code written on its row, when it runs shorter than 120 seconds, carries fewer than 20 samples, contains an internal gap over 30 seconds, lacks real samples at both edges, or has no baseline. It never receives 0.0 joules. A refused block withholds its whole phase rather than dropping out of the sum, because summing the survivors would under-count the refinement phases specifically: their late-round blocks are the small ones that fall under the 120-second floor when only a few tasks are still open, and that is a fail-open in the flattering direction.

3What the pre-registration froze

Five hypotheses, each with a numeric threshold and a stated refutation, all fixed before any run.

Table 1. The five hypotheses. All five are untested. Zero production tasks ran, so none of these was refuted and none was supported.

Ten gates guard the run. G0 is the instrument check, which aborts the orchestrator before any production task. G1 requires the pooled smoke first-pass exact-match to land inside [0.10, 0.90]. G2 requires, per rung, that P(sent back | first pass wrong) minus P(sent back | first pass right) reach 0.20 with at least five tasks in each group, and the campaign stops only when no rung reaches it. G3 needs 0.90 of production blocks to carry a usable wall-joule integral or energy results are withheld. G4 audits duplicates globally across the whole ledger. G5 caps contaminated blocks at 0.20. G6 through G9 cover reproducibility, GPU residency, parse yield and token-budget adequacy.

The calibration rule is the piece that failed, so it is worth quoting exactly. The calibrate stage runs every difficulty level on calibration seeds 9001 to 9005 at horizons {4, 8}, on the bottom and top rungs, and freezes the level whose pooled first-pass exact-match sits closest to 0.50, ties broken by the lower level. The chosen level is written to the campaign's state file before any production task and never revisited. Calibration seeds are 9xxx, smoke seeds 8xxx, production seeds 1xx through 4xx, disjoint by construction.

4The instrument check, and the three defects it caught

G0 ran 75 assertions covering ground truth, grading, task generation, parsing, response fingerprinting, the global distinctness guard, the contamination detector, live model residency, verifier and coordinator behaviour, parse yield, reproducibility and energy. It passed 75 of 75 at 03:42. The run before it failed on one assertion, a missing GPU cross-check log, which is exactly the class of thing a gate is for: the sampler had not been started, and the check refused to proceed rather than reporting a channel it did not have.

Three defects turned up before launch that would each have produced a confident wrong number.

The grader forgave a one-cent error on a task built entirely from cent arithmetic. On a battery whose whole point is exact reconciliation, a grader with a tolerance band scores rounding drift as success. That was found by an assertion that deliberately corrupts the total by a penny and requires exact=False.

The determinism claim was false on real prompts. An early probe on a short prompt returned byte-identical content across repeated calls on every rung, and the pre-registration briefly said the measurement was deterministic on that basis. The instrument check contradicted it on the actual task prompts, and the diagnosis changed the design. The first call after a model load differs from the calls that follow it, which is a prompt-cache effect. Text also changed across a model switch while a second model held memory, and ollama's own log names the mechanism: a partially offloaded model computes some layers on the CPU, and greedy decoding down a different arithmetic path gives a different answer. So byte-identical text was dropped as the claim. What must be stable is what the pilot reports, which is exact-match and token counts, and both are measured directly with thresholds fixed in advance.

The token budget was too small and produced no answer at all. The first frozen budget of 400 + 90n gave 760 tokens at n = 4, and the check found qwen3.5 hitting finish_reason=length with no JSON on 6 of 6 calls, and qwen3:8b on 1 of 6. That is a budget cutting the answer off mid-scratchpad. Resized from measurement to 1,200 + 230n, giving 4,880 tokens at n = 16 against a measured worst case of 3,093, about 1.6 times headroom. G9 was added to check the resizing held in production instead of assuming it.

5The gate outcome

Calibration scored four levels at pooled first-pass exact-match of 0.600, 0.500, 0.400 and 0.450 over 20 tasks each, and froze level 1 at 04:25:43. Smoke then ran 16 tasks on each of two rungs at that level, and the gate read as follows.

Table 2. The smoke gate, 32 outcomes at difficulty level 1. G1 passed on the pooled 0.500 inside [0.10, 0.90]. G2 needs a difference of two conditional rates with at least five tasks in each group, and neither rung produced both groups, so the difference is not estimable for either. The orchestrator wrote SMOKE_GATE_FAILED at 05:40:46 and made no production attempt.

Each rung behaved sensibly taken alone. qwen3:8b sent back 9 of the 16 answers it got wrong, so its checker caught a majority of real defects, and it waved 7 wrong answers straight through on the first look. Two full review rounds moved nothing from wrong to right: final exact-match stayed at 0.000, and mean graded score rose from 0.150 to 0.206, which is partial credit on sub-components rather than a fixed answer. qwen3.5 accepted all 16 of its correct answers and sent back none, which is the right behaviour and the opposite of a fail-open checker, though with no wrong answers to test on there is no evidence separating a good checker from one that accepts everything.

The defect was in the rule, not in the threshold

A pooled first-pass rate of 0.50 arrives just as readily from a floor-and-ceiling pair as from two rungs sitting in the middle, and the pair is the worst arrangement for measurement that a pool of 0.50 can contain. The pooled number is the one that looks like a healthy instrument while carrying no information about whether any single rung can be measured. Level 1 hit it by combining 0 of 16 with 16 of 16.

The fix is structural. Maximise the minimum headroom across rungs, min over rungs of min(p, 1−p), which cannot be satisfied by a floor-and-ceiling split at all. G1's band of [0.10, 0.90] and G2's threshold of 0.20 stay exactly where they were frozen; what changes is the statistic the selection reads.

A second, smaller instrument defect surfaced at the same time. The calibration ledger records parse status, token counts and hashes, and it does not record grades. So the pooled score at each level could not be decomposed by rung after the fact from the ledger alone, and answering the obvious follow-up question required running a fresh diagnostic. A successor should write the grade onto the calibration row.

6Finding one: this host cannot carry metered energy work while its vLLM serves

The top rung is the production vLLM tenant, so a block driving that rung expects the tenant's request counter to move by exactly its own call count, and every other block expects the counter not to move at all. Anything above expectation is a co-tenant request landing mid-measurement. Over the ten smoke blocks, eight were contaminated, a share of 0.80 against G5's ceiling of 0.20, with zero unknown.

One point of bookkeeping before the numbers. The shipped contamination() function is scoped to production blocks and returns an empty result on these, since no production block exists. Everything below applies that function's rule to the smoke and calibration blocks by hand, which is the same arithmetic under a different filter, and it is worth saying rather than implying the gate ran itself.

The calibration stage is the control that makes this readable. All eight of its blocks came back with the counter moving by exactly the number of calls they made, so the co-tenant sat idle through the whole of it. Reconstructing the counter across all eighteen blocks in time order gives a total movement of 207 requests, of which the campaign accounts for 88. The other 119 belong to someone else, and every one of them arrived inside the smoke window. That works out to roughly 1.6 foreign requests a minute over the 74 minutes smoke was running.

Light external traffic contaminates almost everything here, and the mechanism is simple arithmetic. A block is spoiled by one foreign request landing anywhere inside it, and blocks last minutes because a six-second meter needs hundreds of samples to integrate. At an arrival rate of 1.6 a minute, a block sitting at the pre-registered 120-second floor has a 4% chance of staying clean under a Poisson model, and the longer blocks that give better energy resolution do worse. Block-level contamination climbs toward one as the integration window grows, for any non-zero external rate. Energy resolution and exclusivity cannot be optimised together on a shared host.

The same fact, read off the throughput

Contention shows up a second way, in tokens per second, and the calibration stage supplies a clean within-model reference for both rungs.

The sizing consequence is concrete. Smoke measured qwen3.5 at 200.4 seconds per task across its three blocks, with zero repair rounds because it got everything right. Scaling that to the 50 production tasks gives 2.78 hours for that rung with nothing to fix. Estimating the two-round case needs an assumption, and two reasonable ones bracket it: multiplying qwen3.5's first-pass block by the factor qwen3:8b's full two-round loop cost over its own first pass, which is 5.04, gives 7.2 hours, and multiplying the work qwen3.5 completed by the matching 8B subset factor of 2.98 gives 8.3 hours. Add qwen3:8b at 0.71 hours for a full loop over 50 tasks, an estimate for llama3.1:8b of roughly an hour, and the 1.4 hours already spent on gates, and the run lands somewhere between nine and eleven and a half hours against a cap of eleven. The pre-registration sized the whole thing at about 7 hours on throughput measured while the box was quiet.

The design consequence is blunter than a scheduling fix. Exclusive occupancy is a property of the fleet rather than something a campaign can assert about itself, and the pre-registration handled its own exposure well: G5 withholds joules while token results stand, and contaminated blocks leave the energy analysis regardless of whether the gate passes. Nothing degraded silently. What is gone is the joules half, which was the novel part. Future energy work either takes a host out of production for the duration, or runs on a machine that is not in production at all, or models the external load and subtracts it. Subtraction was not available here and should not be retrofitted mid-run, because choosing a contamination model after seeing which blocks are contaminated is what pre-registration exists to prevent.

7Finding two: the battery cannot carry a ladder on this fleet

After the stop, a diagnostic ran the already-burned calibration seeds across all three rungs and all four difficulty levels: 5 seeds by 2 horizons by 4 levels by 3 rungs, 120 tasks. It tests no hypothesis. It answers whether a successor to this battery is feasible at all.

Table 3. The headroom diagnostic, 10 tasks per cell. Headroom is min(p, 1−p), the distance from both floor and ceiling; zero means the cell measures nothing. Had the calibration rule maximised the minimum across rungs, it would have read 0.000 at all four levels and refused to run, saving the smoke stage as well as production.

The 8B rungs are on a floor rather than a low score. Across 80 tasks they produced zero exact matches, while parsing on 0.90 to 1.00 of calls and picking up 0.068 to 0.255 of the graded sub-components. They engage with the task and fail it. That distinction matters, because an unparseable answer would point at a formatting problem the phase-batching could work around, and a parsed answer with the wrong total points at arithmetic these models cannot do to the cent.

This is the third campaign in two days to stop against the same wall, after SIGIL and E10 hard mode on 8 September. The pattern was named in the lab's concept note on the empty difficulty window, and this run adds two things to it. The first is a cost curve: SIGIL found the wall after 2,250 units, E10 after 584, and this pilot after 16, which tracks the instrument checks getting better rather than the fleet getting easier. The second is the diagnostic that would have caught all three at negligible cost, now run and reported: one probe per rung at the intended difficulty, read per rung and never pooled.

A single-rung successor is viable, and the data names where. qwen3.5 at level 2 scores 0.800 exact with headroom 0.200. At 50 tasks that gives about 40 right and 10 wrong, with a 1.9% chance under a binomial that the wrong group falls below G2's floor of five. Level 3 looks close on the surface at 0.900, and the same arithmetic gives it a 43% chance of failing that floor, so the choice between them is not marginal.

8What the smoke ledger holds, and why it is not a test of H1

The smoke stage wrote 131 call rows carrying a phase tag each, which is enough to split tokens by phase. That split is reported here as an observation about 32 tasks at one difficulty level, and it does not test H1. H1 needs a pooled share in tokens and in joules, both above 0.40, over 150 production tasks on three rungs. No joules exist. The smoke rungs sit at floor and ceiling, which is the arrangement most likely to distort a refinement share in both directions at once.

Table 4. Completion tokens by phase, smoke stage, 16 tasks per rung at difficulty level 1. Descriptive only. qwen3:8b got everything wrong and ran every available round, which is the maximum-refinement corner. qwen3.5 got everything right and ran no repair at all, and still spent 40% of its completion tokens on verification.

That last figure is the one worth carrying forward as a question rather than an answer. A rung that needed no repair whatsoever paid four tokens in checking for every ten it spent producing, because the check runs on every task regardless of whether anything is wrong. Counting prompt tokens as well, since a verification prompt carries the answer being checked, the refinement share for qwen3.5 rises to 0.607 and for qwen3:8b to 0.806. Whether those numbers survive on a rung that is neither at floor nor ceiling is the whole question the successor exists to settle.

Coordination is cheap and it is separable, which the design predicted and the ledger confirms. Across all 41 coordination calls in smoke the router spent 1,046 completion tokens, about 25 per routing decision, or 1.8% of the 59,487 completion tokens the stage produced. In the cleanest single block the router made 16 decisions in 12.55 seconds for 385 tokens. Wherever a refinement tax lives on this workflow shape, it is not in the handoff.

9Threats to validity

The difficulty ladder is not monotone

qwen3.5 scores 0.900 at level 3 against 0.800 at level 2, so level 3 is easier for it despite adding a third currency and raising the target rate at which caps bind. The tax rate also differs between them, 0.07375 against 0.0825, which is the obvious suspect and is not established here. A successor should order the ladder by measured difficulty rather than by how hard the knobs look from outside, and should move one knob at a time so a reversal like this can be attributed.

One calibration level does not reproduce, which qualifies a gate this pilot passed

Three of the four levels reproduce exactly between the calibration run at 04:00 and the diagnostic at 06:44: level 1 at 10 of 20, level 2 at 8 of 20, level 3 at 9 of 20. Level 0 recorded a pooled 0.600, meaning 12 of 20, where the diagnostic gives 10 of 20 on identical (seed, horizon, level) inputs at temperature 0.

The calibration ledger records no grades, so that pooled 12 cannot be decomposed from the ledger itself. Attributing the two-task difference to qwen3:8b rests on qwen3.5 having been at ceiling in calibration as it is in the diagnostic and in smoke, which is likely and is not proven. G6 measured exact-match agreement of 1.00 across three interleaved repeats, and it measured that within one model residency. Across a model unload and reload, exact-match is not perfectly stable at the floor. Neither reading changes a conclusion here, since 0 of 10 and 2 of 10 are both floor, and it is recorded because it qualifies a gate that passed.

One further disclosure on G6. Its threshold is the median relative token spread, which came in at 0.000 on both probe rungs. The per-task spreads for qwen3:8b were 0.0 on five of six probe tasks and 0.906 on the sixth. The gate is written on the median and the median passed as specified, and a reader should know that one probe task nearly doubled its token count between repeats.

The throughput comparison mixes two serving stacks

Figure 4 compares blocks within each model and never across them, because qwen3:8b runs on ollama and qwen3.5 on vLLM, and a cross-model tokens-per-second comparison would confound model size with serving stack. Within qwen3.5 the comparison is clean: same model, same phase, same prompt shape, 35.5 against 15.1. Within qwen3:8b the calibration blocks and the smoke first-pass block agree at 38.0 to 38.1 across a stage change and a difficulty change, which is what lets the later drop to 10.8 be read as contention rather than as a harder task.

The contamination detector counts requests, not load

The vLLM counter reports completed requests and says nothing about how large each one was. A single long foreign generation and a single short one both increment it by one. So the contaminated-block share is exact, the foreign request count is exact, and the implied energy contamination is neither, which is precisely why contaminated blocks are excluded from the energy analysis rather than corrected.

The published wiki note on this contamination has two numbers wrong

The lab's concept note on energy measurement needing exclusive occupancy reports calibration contamination at 4 of 8 blocks. Re-derived from the calibration block ledger, every calibration block's counter delta equals its expected delta exactly, so the correct figure is 0 of 8. The 0.50 in that note comes from counting the vLLM rung's own traffic as foreign, which is the error the contamination function's own docstring and the instrument check both warn against. The note also gives smoke as 6 of 8, taken before the last two blocks were written; the final ledger reads 8 of 10.

That note further states that external traffic was about 8% of requests, sourced from a sample of a server log showing 70 requests of 845 arriving from the mini. Nothing in the ledgers supports 8%, and the log it cites is not available on the host now. What the ledgers do support: 119 of the 207 vLLM requests that landed during the campaign window, 57%, did not belong to the campaign, and 111 of those fell inside a measurement block. The conclusion the note draws survives either figure and gets stronger under the ledger's.

10What the single-rung successor should freeze

Eight items, in the order they need deciding.

1. One rung: qwen3.5 at difficulty level 2. Measured first-pass exact-match 0.800, headroom 0.200. At 50 tasks that gives roughly 40 correct and 10 incorrect, both groups clear of G2's floor of five, and the pooled rate sits inside G1's [0.10, 0.90] with room on both sides. Level 3 is rejected on the binomial: at 0.900 the wrong group falls under five about 43% of the time.
2. Calibrate on minimum per-rung headroom. Replace "pooled first-pass exact closest to 0.50" with "maximise min over rungs of min(p, 1−p)", and refuse to run when the maximum is below 0.10. G1's band and G2's 0.20 stay untouched.
3. Record the grade on the calibration row. Exact, graded and parsed, per call. The whole diagnostic in section 7 exists because that column was missing.
4. Drop H2b. A three-point Spearman over two floors and a ceiling carries no information, and running it would put a number in a report that a reader could mistake for evidence. H2a survives intact on task-level rows within one model.
5. Either take a host out of production, or drop the joules claim and say so. Exclusive occupancy is a precondition. Until some host is idle, the honest version of this experiment measures tokens, states that energy was not attempted, and keeps the refusal contract that already works.
6. Keep the block-level refusal contract exactly as written. Error codes in place of zeros, whole phases withheld wherever one block is refused, contaminated blocks excluded regardless of the gate. It is the part of this instrument that behaved perfectly under conditions that broke everything else.
7. Order the difficulty ladder by measured difficulty and move one knob per level. The current ladder inverts between levels 2 and 3 and cannot say why.
8. Keep the phase partition, the router definition, the global distinctness guard and the instrument check unchanged. All four held. The router seeing only the verdict is what makes coordination a measurement rather than a residual, and 25 tokens per decision is the first number this campaign produced that a successor can build on.

The stop is the result. It cost two hours and thirty-eight minutes and it arrived before eleven hours of compute rather than after.