Research paper · campaign E10 · hard-mode headroom pilot
A pre-registered pilot retired an agent experience-curve study on its own kill condition. Verification against the raw ledger reproduces every gate number and shows that the reading which killed it confuses difficulty with answerability.
Campaign E10 asked whether agent operations show a Wright's-law experience
curve, meaning whether fully loaded cost per task falls as a power law in
cumulative volume. Its banked run saturated at a 0.9894 pass rate over 4,830
model attempts, which leaves fewer than 52 failures to fit a curve to, so the
answer carried no information. A pilot raised difficulty using the item
generator's own hard= flag, paired it with a cache key that
generalises over the scan, and pre-registered a kill condition supplied by the
commissioning line: a tranche pass rate outside [0.60, 0.90], or reuse below
0.15, retires E10 permanently.
The pilot ran on 2026-09-08 across 584 attempts with no instrument errors and no fail-open event. Tranche model pass rate came in at 0.5134 (172 of 335, 95% Wilson interval [0.4601, 0.5665]), the kill condition fired, and E10 retires. Independent verification reproduces that number and every other published gate exactly, and it also finds the causal reading unsound.
Hard mode degrades a document by substituting characters drawn from a fixed
table at p = 0.028 per eligible character, applied across the whole
page. The verifier scores invoice_id and po by exact
string equality and the amounts by float comparison, and the ground-truth
dictionary never passes through the noise step. 173 of 400 tranche documents
(43.3%) therefore no longer contain at least one truth value the verifier
demands; in the easy band the count is zero. Restricted to documents whose truth
survives the scan, the tranche reads 0.6701 [0.6012, 0.7324] and the unaided
hard reference block reads 0.8293 [0.6874, 0.9147] against 0.9750 [0.9134,
0.9931] on easy, with overlapping intervals.
The retirement stands, because the gate was pre-registered against a frozen
verifier and that verifier is what ran. Its stated cause does not survive. What the pilot
established is that hard= is an invalid difficulty
instrument: it moves answerability and difficulty at the same time, and roughly
three quarters of the apparent 44-point headroom is documents that cannot be
answered at all.
Wright's law says unit cost falls by a fixed percentage every time cumulative production doubles. E10 asked whether the same shape appears when the producing unit is an agent rather than a factory, with a cache, a retrieval index and a runbook standing in for accumulated experience. Run 2 answered the question on an item set the served model had already saturated. Across 4,830 model attempts the pass rate was 0.9894, which leaves under 52 failures spread over the whole volume axis. Any learning curve fitted there is fitted on noise.
So the card said rebuild at difficulty or retire. The item generator
had carried a hard= flag since
the correlation harness needed one, and the E10 runner had never passed it. The
pilot passed it.
Items are synthetic accounts-payable invoices. The generator builds a
document and a matching truth dictionary, the served model returns extracted
JSON, and a programmatic verifier compares the two field by field. No model
judges another model anywhere in the loop. Serving was the standing vLLM tenant
on spark-train at 127.0.0.1:8001, served-model-name
qwen3.5, reasoning disabled through
chat_template_kwargs and re-confirmed by structural parse at the
start of every stage. The container was read over HTTP and never touched:
StartedAt 2026-08-28T20:14:37Z, RestartCount 0.
Hard mode changes several things at once. Line items go from a mean of 3.52 per document to 9.95, the anomaly rate goes from 0.1125 to 0.5625, total mismatches shrink from dollar scale to cent scale, two awkward tax rates enter the pool, and a scan-degradation pass runs over the finished page. That last step is the subject of this paper.
The frozen E10 cache key is sha1 of the whole document. Under
hard mode a second scan of the same invoice differs from the first on nearly
every line, so that key can never hit a genuine resubmission and reuse would
report zero for reasons unrelated to learning. The pilot added
norm_key, which drops inserted page-furniture lines, collapses
whitespace runs to absorb column drift, and folds the five ambiguous glyph
classes to one member each. Amounts and identifiers survive normalisation
intact, so two different invoices cannot share a key. Instrument-check cases 3
and 4 test both directions.
The pre-registration and every gate and scoring file were committed together
as 0d96b71 at 2026-09-08T17:31:43-05:00. The earliest production
attempt carries the timestamp 17:32:42-05:00. Fifty-nine seconds separate them,
and no threshold was modified afterwards. The kill condition is reproduced
verbatim inside the analysis script as G1 and G2.
| id | quantity | threshold |
|---|---|---|
| G1 | tranche model pass rate | in [0.60, 0.90], kill condition |
| G2 | tranche reuse rate | ≥ 0.15, kill condition |
| G3 | cache efficiency (hits / resubmissions) | ≥ 0.80 |
| G4 | cache precision (hits that verify) | ≥ 0.99 |
| G5 | frozen-key counterfactual reuse | ≤ 0.10 |
| G6 | instrument error rate | ≤ 0.10 |
| G7 | all three mechanisms non-empty | yes |
| H1 | unaided hard below unaided easy | disjoint 95% intervals |
Four stages ran in order: an instrument check on canned items, a 24-item smoke stage with all mechanisms live, a 160-item unaided reference block (80 easy, then 80 hard), and the 400-item tranche the gates are read from. A quarter of tranche attempts are a resubmission, arriving as an independent rescan of an earlier invoice. Both cache keys are evaluated on every attempt, so the counterfactual reuse of the unchanged E10 cache is on the ledger rather than argued for afterwards.
Error handling matters to the reading. A reply that cannot be parsed raises
with a code, the code lands on the record, and passed is set to
null so the attempt leaves the pass-rate denominator entirely. No default answer
is ever substituted for a failed call. Conflating an instrument event with a
model failure is how a broken stack reads as a hard task, and G6 voids the stage
if such events exceed a tenth of it.
Twenty-eight assertions ran before any production attempt, with a non-zero
exit stopping the orchestrator. Two of them assert on meaning rather than
counting records. A cache hit must verify against the truth of the invoice it
was served for, and a live reply to a canned invoice whose subtotal of 250.00
and tax of 0.00 do not reach its TOTAL of 750.00 must come back with
arithmetic_ok: false and anomaly: total_mismatch. Both
are the opposite of what a component skimming or failing open produces. An
independent verification pass later sabotaged six components in memory, among
them a cache returning a wrong entry, a verifier hardwired to return True, and a
generator that ignores hard=, and confirmed the assertions fire.
[PASS] lines. Corrected.| gate | threshold | reading | |
|---|---|---|---|
| G1 tranche model pass rate | in [0.60, 0.90] | 0.5134 [0.4601, 0.5665] n=335 | FAIL |
| G2 reuse rate | ≥ 0.15 | 0.1625 [0.1296, 0.2018] 65/400 | PASS |
| G3 cache efficiency | ≥ 0.80 | 0.6190 65/105 | FAIL |
| G4 cache precision | ≥ 0.99 | 1.0000 65/65 | PASS |
| G5 frozen-key counterfactual | ≤ 0.10 | 0.0000 0/400 | PASS |
| G6 instrument error rate | ≤ 0.10 | 0.0000 0/584 | PASS |
| G7 mechanisms update | all non-empty | cache 172, index 172, runbook 5 | PASS |
| H1 headroom within model | disjoint intervals | 0.9750 vs 0.5375 | PASS as scored |
G1's interval lies entirely under the floor, 0.5665 against 0.60. That is not a borderline reading and the pre-registration forbids retuning toward one. The 65 cache-served attempts sit in their own denominator rather than being excluded from one, so 335 model attempts plus 65 cache hits account for all 400; folding the hits in gives 237/400 = 0.5925 [0.5437, 0.6396], still under the floor.
G3's shortfall follows arithmetically from G1. Forty of the 105 resubmissions missed because the original attempt had failed and was never cached, so at a 0.51 pass rate the highest achievable efficiency is about 0.62, which is what it scored. The 0.80 threshold assumed a pass rate inside G1's band, so with G1 failed, G3 carries no independent meaning.
The harness's OCR noise function walks every character of the finished
document and substitutes from the table {0↔O, 1→l, 5→S,
8→B, .↔,} with probability 0.028 each. It then applies column
drift to lines carrying double spaces and, one document in five, inserts a
page-furniture line. The verifier compares
invoice_id, vendor, currency,
po, line_count, arithmetic_ok and
anomaly by exact equality, and subtotal,
tax and total by float within a cent. The truth
dictionary is built before the noise pass and never sees it.
So the substitution runs over the identifier line and the amount lines that the verifier scores character for character. The generator's own docstring says the ground truth is unchanged and that the invoice still says what it said. That is false as written.
Figure 1 · where the scan sits in the loop
verify() along the dashed path without passing through it, so a substitution inside INV-100040 makes that field unpassable by any reader. Case 1 of the instrument check asserts this split reproduces e10.make_invoice(hard=True) byte for byte over 25 seeds, so the diagram describes the production path.PO-158553 → scan reads PO-15B553PO-513642 → scan reads PO-S13642PO-381618 → scan reads PO-3816l8INV-100040 → scan reads INV-100O40INV-100016 → scan reads INV-l00016INV-100086 → scan reads INV-1000B6
Two criteria appear below because they answer different questions. Value
absent asks whether the exact truth string is missing from the scan, which
is the strict reading of unanswerability. Line altered asks whether any
character on that field's line changed, which also counts damage to the label.
The gap between them comes from the substitution table's keys, which are
0 O 1 5 8 . ,. The words INVOICE, PO: and
TOTAL: each carry an O, so the label can be hit while
the value survives. Subtotal and Tax carry no key
character, which is why both criteria agree on those two counts exactly.
Figure 2 · per-field corruption across 400 tranche documents
po is measured against 355 documents, since 45 carry the missing_po anomaly and have no PO number to damage. The easy band scores zero on every row under either criterion, so this channel is entirely hard-mode specific.Figure 3 · tranche pass rate, split on whether the scan preserved the truth
A caveat belongs on the clean-scan column. Documents with no corrupted scored field also drew fewer substitutions overall, so they are easier to read everywhere on the page, and 0.67 is an optimistic bound rather than an estimate of what a legibility-only instrument would produce.
Figure 4 · four readings of the same 400 attempts
| reading | n | pass rate | inside [0.60, 0.90]? |
|---|---|---|---|
| As pre-registered, frozen verifier | 335 | 0.5134 [0.4601, 0.5665] | no |
invoice_id and po dropped from the verifier | 335 | 0.6209 [0.5678, 0.6712] | yes |
| Restricted to uncorrupted scans, value absent | 194 | 0.6701 [0.6012, 0.7324] | yes |
| Restricted to uncorrupted scans, line altered | 179 | 0.6704 [0.5986, 0.7350] | yes |
The middle reading is the cleaner counterfactual. It conditions on nothing about the document, only on the scoring rule, so it carries no selection bias toward easier pages. Its interval runs from 0.5678 to 0.6712 and straddles the floor.
None of this licenses moving the gate. The pre-registration exists precisely
to stop a scoring rule from being adjusted once the failing fields are visible,
and 36 of the 163 model failures are wrong only on invoice_id
and/or po, which is exactly the shape of evidence that invites the
adjustment. A variant scoring only recoverable fields would be a new
pre-registration and a new decision.
H1 was scored PASS on the raw contrast: easy 0.9750 [0.9134, 0.9931] against hard 0.5375 [0.4290, 0.6425] on the same served model in the same run, with disjoint intervals. The published finding read a 44-point gap off that pair and called it headroom.
Figure 5 · easy against hard, unaided reference block
| band | value absent | line altered |
|---|---|---|
| easy | 78 / 80 = 0.9750 [0.9134, 0.9931] | 78 / 80 = 0.9750 [0.9134, 0.9931] |
| hard | 34 / 41 = 0.8293 [0.6874, 0.9147] | 33 / 38 = 0.8684 [0.7267, 0.9425] |
| drop | 14.6 points | 10.7 points |
Under either criterion the intervals overlap, and H1 says in its own words that it is refuted if they do. The pre-registration also says a drop under ten points will not be claimed; the broader criterion lands at 10.7. Sample size on the hard side falls to about 40 documents once the unanswerable ones are removed, so the effect is real and cannot be pinned down at this scale.
A second confound sits underneath. hard= moves scale, anomaly
rate and legibility together. Even with the answerability problem repaired, a
difference measured between these two bands could not be attributed to reading
difficulty over the near-tripling of line items or the five-fold rise in anomaly
rate.
Anomaly detection is where the model does struggle, and the noise pass is not manufacturing that. Among model attempts whose truth anomaly is null, the model wrongly reported an anomaly on 18 of 51 documents carrying an altered amount line and on 39 of 123 with clean amount lines. The intervals overlap across most of their range.
Figure 6 · anomaly errors and the total_mismatch knob
Twenty-nine documents carrying a genuine total_mismatch reached
the model. The anomaly field was right on five of them and the whole record
passed on three, with discrepancies running from $0.02 to $0.86 and a median of
$0.37. Adding a nine-line invoice to the cent and noticing that it misses by
thirty-seven cents is hard for this model in a way that owes nothing to the
scan.
Reuse decomposes exactly into the stream's resubmission dial times cache efficiency: 0.2625 × 0.6190 = 0.16249, which rounds to the reported 0.1625. The Wilson interval on 65 of 400 runs [0.1296, 0.2018] and contains the 0.15 threshold, so this gate could have gone either way on stream composition alone. The pre-registration disclosed the dial and its reason, and the outcome was already settled by G1, so nothing turns on it. Reporting it as a clean PASS overstates what 65 hits can carry.
G5 asks how often the frozen sha1 of the whole document would
have hit on the same stream. A rescan collides with its original only if the
noise pass draws zero substitutions on both passes. Documents run 400 to 660
characters with a substantial eligible fraction at p = 0.028, so that
event does not occur. G5's 0.000 rules out a trivially duplicated byte stream
and says nothing about whether the generalising key works. The evidence for the
key is G4's 65 of 65 verified hits together with instrument-check cases 3
through 5.
hard=, so no reading in this paper isolates any one of them.invoice_id and po is one repair among several, and it
was chosen because those two fields are pure string identity. Amount fields are
also damaged and were left in.A difficulty knob has to change how hard a document is to read while leaving
the answer recoverable from the page. Three properties would get
hard= there, and none of them requires new science.
PO-15B553.make_invoice_mid.With those in place the E10 question is askable again. It needs a served model sitting somewhere in the middle of its range, and the evidence here says that range exists: 0.975 on easy, roughly 0.83 to 0.87 on hard once answerability is repaired. A generator that lands the model near 0.75 is a tuning problem rather than a research one.
E10 as posed is closed on Dinand's kill condition. Refusing to move the scoring rule after seeing which fields failed is the right call and the whole reason the pre-registration was written.
The claim that hard mode overshoots the usable band has nothing behind it. What the pilot showed is a frozen generator that is invalid as a difficulty instrument, because it manipulates answerability and difficulty together. On the subset where the truth survives the scan, the served model sits near 0.83 on hard against 0.975 on easy, a real and modest effect that 40 documents cannot resolve. The successor campaign gets a new pre-registration, a generator that degrades legibility without destroying recoverability, and an answerability gate that runs before the tranche does.
Every figure in this paper
was derived by replaying the item stream from SEED = 20260908
through the hard-mode generator and its scan pass,
then joining position by position onto the ledger. The replay reproduces
doc_chars, norm_key, exact_key and
resubmit on all 400 tranche and 160 reference records with zero
mismatches, so the corruption analysis is measured on the documents the model
saw rather than on a resampled approximation.