Executive brief · campaign E10 · what the retirement showed
We shut a research programme down on a rule written in advance, which was the right call. Verification then found that the test doing the shutting was marking documents wrong that no longer carried an answer. Both halves of that are worth your attention if you run evaluations.
The question was about cost. Run an AI agent over the same kind of work long enough and does the cost per job fall the way it falls in a factory, roughly a fixed percentage each time cumulative volume doubles? Wright's law has held for aircraft, solar cells and a long list of manufactured things, and nobody had tested it on agent operations. Our first attempt came back flat for a dull reason. The model was already passing 98.9% of items, which leaves about fifty failures spread over five thousand jobs and nothing for a learning curve to bite on.
So we made the work harder. The item generator had a hard mode nobody had ever switched on. We switched it on, wrote down the pass rate the model needed to land in for the experiment to be worth running (between 60% and 90%), committed that number together with all the scoring code, and started the run fifty-nine seconds later. Land outside the band and the programme retires, with no retuning allowed.
It came in at 51%. The programme retired.
Hard mode degrades the page the way a bad scan does. It walks the document and swaps characters that look alike, so a zero becomes an O, a one becomes a lowercase L, a five becomes an S, an eight becomes a B. Realistic enough. The trouble is where it swaps them.
Scoring compares the model's answer against a stored key, character for
character, on fields like the invoice number and the purchase order number. That
key is written before the degradation runs and never passes through it. So when
the generator turns PO-158553 into PO-15B553 on the
page, the page says one thing and the key says another, and a model that reads
the document perfectly is marked wrong.
PO-158553 → page shows PO-15B553INV-100040 → page shows INV-100O40PO-381618 → page shows PO-3816l8
173 of the 400 hard-mode documents, near enough half, had stopped containing at least one value the test demanded. In easy mode that count is zero. The channel switches on with hard mode, which is precisely the comparison the whole study rested on.
Where the damage happens
Four readings of the same 400 items
Take the graded items apart by whether the degradation removed an answer and the split is stark. Documents whose answers survived pass at about two thirds. Documents that lost one pass at about a third. The test could not tell those two populations apart, so it averaged them into a single number and handed us 51%.
Pass rate, split by whether the answer survived
Our published finding said hard mode opened 44 points of headroom, meaning the model dropped from 97.5% on easy items to 53.75% on hard ones. Keep only the hard documents that still carry their own answer and the model scores 83%. The gap narrows to somewhere between eleven and fifteen points, the ranges overlap, and with about forty usable documents there is no pinning it down further. Roughly three quarters of the headroom we reported was documents nobody could have answered.
Easy against hard, before and after removing impossible items
Carry this past the specific study, because the failure reproduces easily and hides well. A difficulty knob is meant to change how hard a question is. Many of the knobs teams reach for change what the question is instead.
Redaction hides the field you grade on. Truncation cuts the passage holding the answer. A paraphrase can drop the named entity an exact-match grader looks for. Ours was character noise over identifiers. Synthetic hard cases written by a second model quietly shift the label. All of these lower the score, a lower score reads as difficulty, and it gets reported as difficulty and acted on.
A score that drops because the answer left the page is indistinguishable from one that drops because the work got harder. The number carries no way to tell them apart.
There is a tell. Failures cluster in the fields the transformation touched. Ours did: 36 of our 163 failures were wrong on nothing except the invoice number and the purchase order number, which are the two pure-string fields the character swaps ran over. If your error breakdown lines up that neatly with your degradation, look at the generator before you conclude anything about the model.
Which fields lost their answer
Before grading anything, ask of each item whether the answer your key demands still appears in the item. No model calls, no cost, a few seconds of compute, and one number out: the share of your evaluation set that is unanswerable as built. Put it beside the score. Gate on it if you want to be strict about it.
Printing that number would have stopped our pilot before the expensive stage ran, for about ninety seconds of work rather than a retired programme and a write-up we had to withdraw.
A second version of the check applies to graders instead of generators. Ask whether your comparison survives the transformation you are applying. Exact string equality does not survive character substitution. Float comparison does not survive a decimal point turning into a comma. Both of those sat in our scoring path, and neither was on anybody's mind, because the scoring code was correct on its own and so was the generator. The defect lived in the seam between them, which is where this class of defect usually lives.
This correction exists at all because of the rule we wrote down first.
When the number came back at 51%, we had in hand a breakdown showing which fields had failed and a very reasonable-sounding case that those failures were an artefact. Drop the two damaged fields from the scoring and the pass rate is 62%, inside the band, and the programme survives. Nobody outside the team would have known.
We left the rule alone and the programme stayed retired. The reason is worth saying plainly. That adjustment only became visible after we saw which fields had failed, and an adjustment that becomes available only once you know the result is a search for a scoring rule that gives you the answer you wanted. There is always one available. Someone will find it.
Pre-registration is only doing work when it holds a conclusion you would rather move.
What we bought is that the correction, when it arrived, was about the instrument rather than about the verdict. We kept the retirement and threw away our explanation of it, and those two things could be separated only because the rule had been frozen beforehand. The whole apparatus was committed in one go a minute before the first call went out, which is what makes the claim checkable by someone who does not trust us.
The retirement stands. The underlying question survives it, because the evidence now says the room to measure a learning curve is there: 97.5% on easy items, around 83% on hard ones once the impossible items come out. A generator that lands the model near 75% is a tuning job rather than a research problem.
The successor needs a difficulty knob that degrades legibility while leaving the answer recoverable from the page, which the current one demonstrably does not do. It also needs to move one dial at a time. Hard mode currently triples the line count, raises the anomaly rate five-fold and blurs the characters all at once, so even with the answerability defect repaired, no measured drop could be attributed to any single cause. That is a new pre-registration and a new decision.
One finding did survive intact, and it is the useful part of the wreckage. The model's real weakness is arithmetic on a degraded page: given a nine-line invoice whose total is off by a median of 37 cents, it caught the discrepancy 5 times out of 29. Nothing about the character swapping causes that. Cent-scale reconciliation on long documents is a real, measurable, non-saturated task, and it is where the next version of this study should point.