# Difficulty That Removes the Answer Source: https://dinand.com/research/difficulty-that-removes-the-answer/ Author: Dinand Tinholt Published: 2026-09-08 > A pilot retired on its own kill condition, but 43.3% of its hard documents no longer carried the answer, which retracts the published 44 points of headroom. Difficulty That Removes the Answer Research paper · campaign E10 · hard-mode headroom pilot # Difficulty That Removes the Answer A pre-registered pilot retired an agent experience-curve study on its own kill condition. Verification against the raw ledger reproduces every gate number and shows that the reading which killed it confuses difficulty with answerability. 584 attempts · 2026-09-08, 17:32–18:02 CDT, spark-train · zero instrument errors · pre-registration frozen 59 seconds before the first production call · every figure below re-derived by deterministic replay of the item stream Abstract Campaign E10 asked whether agent operations show a Wright's-law experience curve, meaning whether fully loaded cost per task falls as a power law in cumulative volume. Its banked run saturated at a 0.9894 pass rate over 4,830 model attempts, which leaves fewer than 52 failures to fit a curve to, so the answer carried no information. A pilot raised difficulty using the item generator's own `hard=` flag, paired it with a cache key that generalises over the scan, and pre-registered a kill condition supplied by the commissioning line: a tranche pass rate outside [0.60, 0.90], or reuse below 0.15, retires E10 permanently. The pilot ran on 2026-09-08 across 584 attempts with no instrument errors and no fail-open event. Tranche model pass rate came in at 0.5134 (172 of 335, 95% Wilson interval [0.4601, 0.5665]), the kill condition fired, and E10 retires. Independent verification reproduces that number and every other published gate exactly, and it also finds the causal reading unsound. Hard mode degrades a document by substituting characters drawn from a fixed table at p = 0.028 per eligible character, applied across the whole page. The verifier scores `invoice_id` and `po` by exact string equality and the amounts by float comparison, and the ground-truth dictionary never passes through the noise step. 173 of 400 tranche documents (43.3%) therefore no longer contain at least one truth value the verifier demands; in the easy band the count is zero. Restricted to documents whose truth survives the scan, the tranche reads 0.6701 [0.6012, 0.7324] and the unaided hard reference block reads 0.8293 [0.6874, 0.9147] against 0.9750 [0.9134, 0.9931] on easy, with overlapping intervals. The retirement stands, because the gate was pre-registered against a frozen verifier and that verifier is what ran. Its stated cause does not survive. What the pilot established is that `hard=` is an invalid difficulty instrument: it moves answerability and difficulty at the same time, and roughly three quarters of the apparent 44-point headroom is documents that cannot be answered at all. Tranche pass rate, as pre-registered 0.5134 Documents missing a scored truth value 43.3% Same documents, easy band 0 / 80 Hard pass rate where the truth survives 0.8293 ## 1. What E10 asked, and why the first answer was empty Wright's law says unit cost falls by a fixed percentage every time cumulative production doubles. E10 asked whether the same shape appears when the producing unit is an agent rather than a factory, with a cache, a retrieval index and a runbook standing in for accumulated experience. Run 2 answered the question on an item set the served model had already saturated. Across 4,830 model attempts the pass rate was 0.9894, which leaves under 52 failures spread over the whole volume axis. Any learning curve fitted there is fitted on noise. So the card said rebuild at difficulty or retire. The item generator had carried a `hard=` flag since the correlation harness needed one, and the E10 runner had never passed it. The pilot passed it. ## 2. The apparatus Items are synthetic accounts-payable invoices. The generator builds a document and a matching truth dictionary, the served model returns extracted JSON, and a programmatic verifier compares the two field by field. No model judges another model anywhere in the loop. Serving was the standing vLLM tenant on spark-train at `127.0.0.1:8001`, served-model-name `qwen3.5`, reasoning disabled through `chat_template_kwargs` and re-confirmed by structural parse at the start of every stage. The container was read over HTTP and never touched: `StartedAt` 2026-08-28T20:14:37Z, `RestartCount` 0. Hard mode changes several things at once. Line items go from a mean of 3.52 per document to 9.95, the anomaly rate goes from 0.1125 to 0.5625, total mismatches shrink from dollar scale to cent scale, two awkward tax rates enter the pool, and a scan-degradation pass runs over the finished page. That last step is the subject of this paper. ### The cache key The frozen E10 cache key is `sha1` of the whole document. Under hard mode a second scan of the same invoice differs from the first on nearly every line, so that key can never hit a genuine resubmission and reuse would report zero for reasons unrelated to learning. The pilot added `norm_key`, which drops inserted page-furniture lines, collapses whitespace runs to absorb column drift, and folds the five ambiguous glyph classes to one member each. Amounts and identifiers survive normalisation intact, so two different invoices cannot share a key. Instrument-check cases 3 and 4 test both directions. ## 3. Pre-registration The pre-registration and every gate and scoring file were committed together as `0d96b71` at 2026-09-08T17:31:43-05:00. The earliest production attempt carries the timestamp 17:32:42-05:00. Fifty-nine seconds separate them, and no threshold was modified afterwards. The kill condition is reproduced verbatim inside the analysis script as G1 and G2. | id | quantity | threshold | | --- | --- | --- | | G1 | tranche model pass rate | in [0.60, 0.90], kill condition | | G2 | tranche reuse rate | ≥ 0.15, kill condition | | G3 | cache efficiency (hits / resubmissions) | ≥ 0.80 | | G4 | cache precision (hits that verify) | ≥ 0.99 | | G5 | frozen-key counterfactual reuse | ≤ 0.10 | | G6 | instrument error rate | ≤ 0.10 | | G7 | all three mechanisms non-empty | yes | | H1 | unaided hard below unaided easy | disjoint 95% intervals | Four stages ran in order: an instrument check on canned items, a 24-item smoke stage with all mechanisms live, a 160-item unaided reference block (80 easy, then 80 hard), and the 400-item tranche the gates are read from. A quarter of tranche attempts are a resubmission, arriving as an independent rescan of an earlier invoice. Both cache keys are evaluated on every attempt, so the counterfactual reuse of the unchanged E10 cache is on the ledger rather than argued for afterwards. Error handling matters to the reading. A reply that cannot be parsed raises with a code, the code lands on the record, and `passed` is set to null so the attempt leaves the pass-rate denominator entirely. No default answer is ever substituted for a failed call. Conflating an instrument event with a model failure is how a broken stack reads as a hard task, and G6 voids the stage if such events exceed a tenth of it. ### The instrument check Twenty-eight assertions ran before any production attempt, with a non-zero exit stopping the orchestrator. Two of them assert on meaning rather than counting records. A cache hit must verify against the truth of the invoice it was served for, and a live reply to a canned invoice whose subtotal of 250.00 and tax of 0.00 do not reach its TOTAL of 750.00 must come back with `arithmetic_ok: false` and `anomaly: total_mismatch`. Both are the opposite of what a component skimming or failing open produces. An independent verification pass later sabotaged six components in memory, among them a cache returning a wrong entry, a verifier hardwired to return True, and a generator that ignores `hard=`, and confirmed the assertions fire. The published finding originally said thirty assertions. The log carries twenty-eight `[PASS]` lines. Corrected. ## 4. Results | gate | threshold | reading | | | G1 tranche model pass rate | in [0.60, 0.90] | 0.5134 [0.4601, 0.5665] n=335 | FAIL | | G2 reuse rate | ≥ 0.15 | 0.1625 [0.1296, 0.2018] 65/400 | PASS | | G3 cache efficiency | ≥ 0.80 | 0.6190 65/105 | FAIL | | G4 cache precision | ≥ 0.99 | 1.0000 65/65 | PASS | | G5 frozen-key counterfactual | ≤ 0.10 | 0.0000 0/400 | PASS | | G6 instrument error rate | ≤ 0.10 | 0.0000 0/584 | PASS | | G7 mechanisms update | all non-empty | cache 172, index 172, runbook 5 | PASS | | H1 headroom within model | disjoint intervals | 0.9750 vs 0.5375 | PASS as scored | G1's interval lies entirely under the floor, 0.5665 against 0.60. That is not a borderline reading and the pre-registration forbids retuning toward one. The 65 cache-served attempts sit in their own denominator rather than being excluded from one, so 335 model attempts plus 65 cache hits account for all 400; folding the hits in gives 237/400 = 0.5925 [0.5437, 0.6396], still under the floor. G3's shortfall follows arithmetically from G1. Forty of the 105 resubmissions missed because the original attempt had failed and was never cached, so at a 0.51 pass rate the highest achievable efficiency is about 0.62, which is what it scored. The 0.80 threshold assumed a pass rate inside G1's band, so with G1 failed, G3 carries no independent meaning. ## 5. The confound The harness's [OCR noise function](https://dinand.com/research/studies-the-gates-stopped/#ocr-correction) walks every character of the finished document and substitutes from the table `{0↔O, 1→l, 5→S, 8→B, .↔,}` with probability 0.028 each. It then applies column drift to lines carrying double spaces and, one document in five, inserts a page-furniture line. The verifier compares `invoice_id`, `vendor`, `currency`, `po`, `line_count`, `arithmetic_ok` and `anomaly` by exact equality, and `subtotal`, `tax` and `total` by float within a cent. The truth dictionary is built before the noise pass and never sees it. So the substitution runs over the identifier line and the amount lines that the verifier scores character for character. The generator's own docstring says the ground truth is unchanged and that the invoice still says what it said. That is false as written. Figure 1 · where the scan sits in the loop *The noise pass runs on the document only. The truth dictionary reaches `verify()` along the dashed path without passing through it, so a substitution inside `INV-100040` makes that field unpassable by any reader. Case 1 of the instrument check asserts this split reproduces `e10.make_invoice(hard=True)` byte for byte over 25 seeds, so the diagram describes the production path.* Verified renderings, replayed from the tranche stream truth `PO-158553` → scan reads PO-15B553 truth `PO-513642` → scan reads PO-S13642 truth `PO-381618` → scan reads PO-3816l8 truth `INV-100040` → scan reads INV-100O40 truth `INV-100016` → scan reads INV-l00016 truth `INV-100086` → scan reads INV-1000B6 ### 5.1 How often Two criteria appear below because they answer different questions. *Value absent* asks whether the exact truth string is missing from the scan, which is the strict reading of unanswerability. *Line altered* asks whether any character on that field's line changed, which also counts damage to the label. The gap between them comes from the substitution table's keys, which are `0 O 1 5 8 . ,`. The words `INVOICE`, `PO:` and `TOTAL:` each carry an `O`, so the label can be hit while the value survives. `Subtotal` and `Tax` carry no key character, which is why both criteria agree on those two counts exactly. Figure 2 · per-field corruption across 400 tranche documents *`po` is measured against 355 documents, since 45 carry the `missing_po` anomaly and have no PO number to damage. The easy band scores zero on every row under either criterion, so this channel is entirely hard-mode specific.* ### 5.2 What it does to the pass rate Figure 3 · tranche pass rate, split on whether the scan preserved the truth *The two criteria partition the same 335 model attempts differently and agree closely on the clean side. A document whose scored truth survived the scan passes at roughly 0.67, inside the pre-registered band. One that lost a scored value passes at roughly 0.30.* A caveat belongs on the clean-scan column. Documents with no corrupted scored field also drew fewer substitutions overall, so they are easier to read everywhere on the page, and 0.67 is an optimistic bound rather than an estimate of what a legibility-only instrument would produce. ## 6. Three readings of one tranche Figure 4 · four readings of the same 400 attempts *The pre-registered reading is the one that governs, and it sits clear of the band. The corrected readings all move inside, and every one of their intervals still straddles the 0.60 floor, so none of them is evidence that the gate would have passed on a valid instrument.* | reading | n | pass rate | inside [0.60, 0.90]? | | As pre-registered, frozen verifier | 335 | 0.5134 [0.4601, 0.5665] | no | | `invoice_id` and `po` dropped from the verifier | 335 | 0.6209 [0.5678, 0.6712] | yes | | Restricted to uncorrupted scans, value absent | 194 | 0.6701 [0.6012, 0.7324] | yes | | Restricted to uncorrupted scans, line altered | 179 | 0.6704 [0.5986, 0.7350] | yes | The middle reading is the cleaner counterfactual. It conditions on nothing about the document, only on the scoring rule, so it carries no selection bias toward easier pages. Its interval runs from 0.5678 to 0.6712 and straddles the floor. None of this licenses moving the gate. The pre-registration exists precisely to stop a scoring rule from being adjusted once the failing fields are visible, and 36 of the 163 model failures are wrong only on `invoice_id` and/or `po`, which is exactly the shape of evidence that invites the adjustment. A variant scoring only recoverable fields would be a new pre-registration and a new decision. ## 7. H1 does not survive restriction H1 was scored PASS on the raw contrast: easy 0.9750 [0.9134, 0.9931] against hard 0.5375 [0.4290, 0.6425] on the same served model in the same run, with disjoint intervals. The published finding read a 44-point gap off that pair and called it headroom. Figure 5 · easy against hard, unaided reference block *Restricting the hard band to documents whose scored truth survived the scan moves it from 0.5375 to between 0.8293 and 0.8684 depending on criterion. Both restricted intervals overlap the easy interval, which is H1's own stated refutation condition.* | band | value absent | line altered | | easy | 78 / 80 = 0.9750 [0.9134, 0.9931] | 78 / 80 = 0.9750 [0.9134, 0.9931] | | hard | 34 / 41 = 0.8293 [0.6874, 0.9147] | 33 / 38 = 0.8684 [0.7267, 0.9425] | | drop | 14.6 points | 10.7 points | Under either criterion the intervals overlap, and H1 says in its own words that it is refuted if they do. The pre-registration also says a drop under ten points will not be claimed; the broader criterion lands at 10.7. Sample size on the hard side falls to about 40 documents once the unanswerable ones are removed, so the effect is real and cannot be pinned down at this scale. A second confound sits underneath. `hard=` moves scale, anomaly rate and legibility together. Even with the answerability problem repaired, a difference measured between these two bands could not be attributed to reading difficulty over the near-tripling of line items or the five-fold rise in anomaly rate. ## 8. What survives as genuine difficulty Anomaly detection is where the model does struggle, and the noise pass is not manufacturing that. Among model attempts whose truth anomaly is null, the model wrongly reported an anomaly on 18 of 51 documents carrying an altered amount line and on 39 of 123 with clean amount lines. The intervals overlap across most of their range. Figure 6 · anomaly errors and the total_mismatch knob *False anomaly reports run at about the same rate whether or not the amount lines were damaged, so the 116 anomaly failures are not an artefact of the scan. Cent-scale total mismatches are the sharpest real difficulty dial in the set.* Twenty-nine documents carrying a genuine `total_mismatch` reached the model. The anomaly field was right on five of them and the whole record passed on three, with discrepancies running from $0.02 to $0.86 and a median of $0.37. Adding a nine-line invoice to the cent and noticing that it misses by thirty-seven cents is hard for this model in a way that owes nothing to the scan. ## 9. Two gates that need a caveat ### G2 is a point estimate Reuse decomposes exactly into the stream's resubmission dial times cache efficiency: 0.2625 × 0.6190 = 0.16249, which rounds to the reported 0.1625. The Wilson interval on 65 of 400 runs [0.1296, 0.2018] and contains the 0.15 threshold, so this gate could have gone either way on stream composition alone. The pre-registration disclosed the dial and its reason, and the outcome was already settled by G1, so nothing turns on it. Reporting it as a clean PASS overstates what 65 hits can carry. ### G5 cannot fail under hard mode G5 asks how often the frozen `sha1` of the whole document would have hit on the same stream. A rescan collides with its original only if the noise pass draws zero substitutions on both passes. Documents run 400 to 660 characters with a substantial eligible fraction at p = 0.028, so that event does not occur. G5's 0.000 rules out a trivially duplicated byte stream and says nothing about whether the generalising key works. The evidence for the key is G4's 65 of 65 verified hits together with instrument-check cases 3 through 5. ## 10. Threats to validity 1. Selection on the restricted readings. Documents with no corrupted scored field drew fewer substitutions in total, so they are easier throughout. 0.6701 and 0.8293 are upper bounds on what a legibility-only instrument would show. 2. Confounded bands. Line count, anomaly rate and legibility all move with `hard=`, so no reading in this paper isolates any one of them. 3. Single served model. Everything here is qwen3.5 on one tenant. The banked 0.9894 came from a different served model on spark-inference, which is why the within-run reference block exists at all. 4. One tranche. No slope is fitted anywhere. The pre-registration explicitly excludes a Wright's-law fit, since a single tranche cannot show one, and the first E10 run's flat curve came from fitting across 42 processes that each started with an empty cache. 5. Small restricted samples. The answerable hard reference block holds 38 to 41 documents. Interval half-widths there run about 12 points. 6. The verifier's fields are not equally recoverable. Dropping `invoice_id` and `po` is one repair among several, and it was chosen because those two fields are pure string identity. Amount fields are also damaged and were left in. ## 11. What a valid successor instrument needs A difficulty knob has to change how hard a document is to read while leaving the answer recoverable from the page. Three properties would get `hard=` there, and none of them requires new science. 1. Keep scored tokens out of the substitution domain, or carry the rendered value into the truth so the verifier scores what the page itself says. Either repair makes every item answerable by construction. The second is closer to how a real OCR benchmark works, since a human reading the scan would also read `PO-15B553`. 2. Move one dial at a time. Line count, anomaly rate and legibility need separate flags so a measured drop can be attributed. A mid-difficulty setting already exists in corr2 as `make_invoice_mid`. 3. Report an answerability audit beside the pass rate. The count of items whose truth value is absent from the item is cheap to compute, needs no model calls, and would have caught this before the tranche ran. A pilot that prints it as a gate cannot mistake an impossible item for a hard one. With those in place the E10 question is askable again. It needs a served model sitting somewhere in the middle of its range, and the evidence here says that range exists: 0.975 on easy, roughly 0.83 to 0.87 on hard once answerability is repaired. A generator that lands the model near 0.75 is a tuning problem rather than a research one. ## 12. What retires, and what does not E10 as posed is closed on Dinand's kill condition. Refusing to move the scoring rule after seeing which fields failed is the right call and the whole reason the pre-registration was written. The claim that hard mode overshoots the usable band has nothing behind it. What the pilot showed is a frozen generator that is invalid as a difficulty instrument, because it manipulates answerability and difficulty together. On the subset where the truth survives the scan, the served model sits near 0.83 on hard against 0.975 on easy, a real and modest effect that 40 documents cannot resolve. The successor campaign gets a new pre-registration, a generator that degrades legibility without destroying recoverability, and an answerability gate that runs before the tranche does. ## Reproduction Every figure in this paper was derived by replaying the item stream from `SEED = 20260908` through the hard-mode generator and its scan pass, then joining position by position onto the ledger. The replay reproduces `doc_chars`, `norm_key`, `exact_key` and `resubmit` on all 400 tranche and 160 reference records with zero mismatches, so the corruption analysis is measured on the documents the model saw rather than on a resampled approximation. --- Site guide for AI agents: https://dinand.com/llms.txt