# Shared Training Data Does Not Correlate Errors

Source: https://dinand.com/research/shared-training-data/
Author: Dinand Tinholt
Published: 2026-09-08

> Two models that share a training corpus correlate their errors no more than unrelated models do (+0.027, 95% interval −0.164 to +0.206), and the earlier shared-weights result holds once defective documents are removed.

Shared Training Data Does Not Correlate Errors

Pre-registered study · 7 models · 150 items · 1,050 scored calls

# Shared training data does not correlate errors

Two deployments of identical weights on different silicon fail on the same items. An earlier study measured that at phi +0.447, against -0.102 for pairs drawn from different model families, and left one question hanging: what if two models share a training corpus and no weights at all? This run answers it. The shared-corpus pair lands at +0.149. The cross-family controls beside it average +0.122. The pre-registered interval on that gap runs from -0.164 to +0.206, so the hypothesis is refuted by its own rule. Late in the write-up I found that 16% of the items ask for a string the document does not contain, so every headline number below is given twice, once as registered and once with those items removed. The refutation holds under both and gets firmer under the second.

Pre-registered 2026-09-06, amended twice before any full-run call · run complete 2026-09-08 12:54 UTC, RUN_EXIT_0 · every number below recomputed from the raw per-call ledger by an independent parser

+0.149

conditioned phi for the shared-corpus pair

+0.122

mean over the ten cross-family control pairs

+0.027

the registered contrast, interval −0.164 to +0.206

107

contested items surviving difficulty conditioning

6 of 15

where the treatment pair ranks among all pairs

## What the earlier work left open

The published study measured pairwise error correlation on an invoice extraction task. Ground truth comes from the generator that writes the invoice, so no model ever grades another and there is no judge to be biased. Each model scores the same items, an error indicator is 1 where a model gets an item wrong, and the phi coefficient runs over those indicators for every pair.

Raw phi looked uniformly high. Conditioning on item difficulty overturned that: drop the items every model got right and the items every model got wrong, because those carry no information about whether two models fail together, and recompute on what is left. On the 47 items that survived, two deployments of identical weights on different hardware sat at +0.447 while the cross-family pairs averaged -0.102. I re-executed the frozen analysis against the archived error vectors before starting this run and got +0.4475 and -0.1015 on 47 items, which is the published pair of numbers to three decimals.

That result says independence tracks the model family. The mechanism behind it stayed open. Weight sharing is one channel by which two systems could acquire the same blind spot. A shared training corpus is a second channel, and this one can be isolated. DeepSeek fine-tuned the same set of roughly 800k reasoning traces onto base models drawn from unrelated pretraining runs, so two of those distills share every training sample in the fine-tune and share nothing at all in their weights. That gives the experiment without anything needing to be built.

## What was registered, and when

The pre-registration was written down on 2026-09-06, before any experimental call went out. It fixed the hypothesis with a direction, the pair taxonomy, the decision rule and the bootstrap, and it committed in writing to publishing a refutation.

>

H1. Shared training signal without shared weights raises difficulty-conditioned phi above the cross-family baseline, toward the same-weights value. Concretely: contested-item phi for the treatment pair exceeds the mean contested-item phi over the cross-family pairs of this run.

Decision rule. Delta = phi(T) minus mean phi over the X pairs, on contested items, with a 10,000-resample bootstrap over those items for the 95% interval. Supported when Delta > 0 and the interval excludes zero. Refuted when the interval includes zero and the point estimate falls below +0.15. Inconclusive otherwise.

Two amendments followed, both written after a pilot and both dated before any full-run call was made. Amendment 1 dealt with a capability problem. The original treatment pair put a 32B distill against an 8B one, and on the pilot the 8B rungs answered nothing correctly at all. A model at ceiling error has no variance, and phi is undefined against it. So the primary treatment pair was re-designated to the 70B distill built on a Llama 3.3 base against the 32B distill built on a Qwen2.5 base. Same fine-tune corpus, unrelated pretraining, and now matched to the difficulty of the task.

Amendment 2 is longer and it changes two things that matter for reading the numbers. Both are disclosed in full below.

## The item set is no longer the published one

The Amendment-1 pilot floored every model on the hard item set. Hand inspection of the raw outputs showed the failures were real rather than a harness artifact. That reading was wrong, and the section "Some items do not contain the answer they are scored against" below replaces it. The models returned well-formed JSON that transcribed the OCR-noised invoice id verbatim where the truth is the clean value, and that miscounted long line blocks badly. A dial sweep established that line count rather than OCR rate is the lever for the smaller models, and the item set was retuned to 4 to 8 lines, mismatch magnitude 0.5 to 40, OCR rate 0.01, anomaly rate 0.45. Every other generator parameter is the original one, verbatim, down to the seed.

I regenerated both item sets from scratch during verification. The retuned set hashes to `4b89fdbc893a701d…` and the published set to `6db1ef80b3a1d143…`, both matching what the pre-registration pinned. The retuned set came out at 86 clean invoices, 13 with a total mismatch, 13 with a missing purchase order, 12 with a duplicated line, 10 credit notes and 16 carrying a second currency, which matches what the pre-registration listed, item for item.

The consequence, stated plainly. This study runs on a different item set from the published one. Laying its phi values alongside the published +0.447 and -0.102 as though they sat on one scale would be wrong. Cross-study comparison here is indicative only. The pre-registered test is a comparison made *within* this run, between the treatment pair and the control pairs scored on the same items by the same harness, and that comparison is untouched by the item-set change.

How much the scale moved is measurable. This run's cross-family controls average +0.122 on contested items where the published run's averaged -0.102. Most of that shift comes from the width of the contested net rather than from the items themselves: on the tightest usable band, which keeps 39% of items against the published run's 31%, this run's cross-family mean falls to -0.059.

## The programmatic juror in the published run was broken

Triaging the pilot exposed a bug that is verbatim in the published harness. The deterministic checker, included as a non-model juror and an independence floor, searched for its labels case-insensitively. The pattern for `TOTAL` therefore matched inside `Subtotal:` first and grabbed the wrong number. On 147 of the 150 published items the match landed inside the word "Subtotal", after which the juror tested whether subtotal plus tax equals the subtotal. That holds only where tax is zero, which was true of 25 of those items.

Model scoring never touched that code path. It runs through the generator's own parse and verify functions, which are separate, so no published model-pair phi moves and the headline of the earlier study survives intact. The sentence about the independence floor needs withdrawing. The published juror scored 14 out of 150 and showed phi between -0.155 and -0.012 against every model, and those numbers recompute exactly from the archived vectors. They describe a juror that was answering close to at random.

Both jurors ran here. The repaired one anchors its labels to the start of a line, keeps case, and maps the OCR-noised comma back to a decimal point. It scores 115 of 150 on this study's items and 56 of 150 on the published ones. The original is retained under the name `det_check_v1`, reported for continuity at 28 of 150 here and 14 of 150 on the published set, and quoted nowhere as a result. Neither juror enters any model-pair phi and neither enters the difficulty vector, so nothing in this study rests on the broken vectors. I checked the counterfactual as well: forcing the repaired juror into the difficulty vector moves the headline contrast to +0.004, and forcing the broken one in moves it to -0.037.

## Some items do not contain the answer they are scored against

The noise function walks every character of the document and swaps 0 for O, 1 for l, 5 for S, 8 for B, a decimal point for a comma. It does that wherever a matching character sits. An invoice id is a run of digits, so it gets the same treatment as the vendor address. Then `verify` scores `invoice_id` and `po` by exact string equality, and the three amounts by float comparison, against values the generator wrote down before the noise ran.

So on some items the harness hands a model a line reading `INVOICE INV-1O0114` and then marks it wrong for failing to answer `INV-100114`. The docstring on that function says the ground truth is unchanged and the invoice is only harder to read. The truth dictionary is unchanged. The document has moved.

I wrote earlier in this article that hand inspection of the pilot outputs showed the failures were real rather than a harness artifact. I withdraw that. The models were transcribing what the page said, which is the job, and the harness was asking for a string the page no longer carried.

Rebuilding the pinned item set and aligning every noised line back to the clean line it came from puts the damage at 24 of 150 items, 16.0%, where at least one scored field has a character substituted inside the value.

| scored field | items where the truth string is absent | example |
| --- | --- | --- |
| invoice_id | 10 | `INV-l00054` for `INV-100054` |
| po | 5 | `PO-1B9047` for `PO-189047` |
| subtotal | 4 | `3255l.36` for `32551.36` |
| tax | 3 | `5420.S7` for `5420.57` |
| total | 3 | `2164S.36` for `21645.36` |

Items 5, 6, 23, 25, 26, 44, 54, 56, 64, 65, 69, 71, 72, 75, 76, 80, 85, 92, 109, 114, 126, 134, 136 and 137. Item 26 carries damage in two fields at once, its purchase order and its subtotal, which is why the column sums to 25 across 24 items.

Three further items have an amount whose decimal point became a comma with every digit intact, `50555,60` against a truth of `50555.60`. I hold those apart, since anyone who has read a European invoice loses nothing there. Counting them takes the total to 27.

Calling these items unanswerable would overstate it. Inside a number the substitution inverts cleanly, because none of the replacement characters is a digit, so `51l96.50` can only have come from `51196.50`. Some models do invert it. Item 92 carries a corrupted tax digit and all six gate-passing models scored it correct. What the items reliably do is put the demanded string out of reach of a straight read, and models fail them at rates they hit nowhere else in the set.

| model | error rate on the 24 | on the other 126 | lift |
| r1-70b-llama3.3 (treatment) | 87.5% | 13.5% | +0.740 |
| qwen3-30b-a3b | 75.0% | 6.3% | +0.687 |
| r1-32b-qwen2.5 (treatment) | 91.7% | 29.4% | +0.623 |
| gemma3-27b | 75.0% | 16.7% | +0.583 |
| qwen3-8b | 83.3% | 31.0% | +0.524 |
| llama3.1-8b | 95.8% | 76.2% | +0.196 |
| r1-8b-llama3.1 (failed the gate) | 100% | 98.4% | +0.016 |
| repaired programmatic juror | 70.8% | 14.3% | +0.565 |

The two small lifts are ceilings. Both of those models were already wrong on most of the set, so there was nothing left to give. The lift is spread across every family that had room to move, and the treatment models are not unusual within it.

The pre-registered difficulty rule had been catching most of this without anyone knowing. Sixteen items were failed by every gate-passing model and dropped as unanimous. All sixteen sit inside the 24. No answerable item in the set was failed by all six. That rule exists to drop items carrying no information about whether two systems agree for a shared reason, and what it was quietly removing was the broken ones.

Seven of the 24 reached the contested pool and one was passed unanimously. Seven items out of 107 turn out to be enough to move the headline, because on five of them both treatment models fail together, which is the cell that phi(T) is built out of.

| reading | items | contested | phi(T) | mean phi(X) | Delta | 95% interval |
| as registered | 150 | 107 | +0.149 | +0.122 | +0.027 | −0.164 to +0.206 |
| minus the 24, roster held | 126 | 100 | +0.039 | +0.114 | −0.074 | −0.268 to +0.113 |
| minus the 24, gate re-applied | 126 | 96 | −0.094 | +0.073 | −0.167 | −0.355 to +0.033 |

Both cleaned readings, with the same 10,000-resample bootstrap. The roster matters because on the cleaned pool the Qwen3 MoE falls to 8 errors in 126, which is 6.3% and below the pre-registered 10% floor. Holding the roster keeps the comparison like for like. Re-applying the gate is the letter of the registration and drops that model, which strips three control pairs out of the X cell.

Every reading refutes H1 by the registered rule, and the cleaned ones refute it harder. The defect was pushing the treatment pair up, so removing it moves the answer further from the hypothesis. Delta was +0.027 against a refutation threshold of +0.15 and it becomes negative. The treatment pair slides from sixth of fifteen to tenth. The one mildly suggestive number in the whole run, 12 joint failures against 8.8 expected under independence at a permutation p of 0.096, becomes 7 against 6.3 expected at p = 0.449, which is nothing at all.

Raw phi is where the real damage sits. Read on all 150 items with no conditioning, the three cells came out at +0.410, +0.425 and +0.426. Drop the 24 and they read +0.102, +0.230 and +0.241. Roughly two thirds of the raw correlation in the treatment cell was manufactured by items nobody could answer. Anyone tempted to quote a raw phi off a noised item set should look at that pair of rows first.

The juror is the cleanest demonstration. It is a regular expression. It has no weights and no training corpus, and it cannot share a blind spot with a language model. Its `grab()` reads amounts with `([\d.,]+)`, which will not match a token holding an O or an l, so the pattern returns nothing and the arithmetic test collapses. Its id pattern anchors on the literal word `INVOICE`. Seventeen of its 35 errors fall on the 24 items, a 70.8% error rate against 14.3% everywhere else, which is the whole of its apparent correlation with a 70B model.

Which means the independence-floor reading later in this article does not survive, and I have rewritten that section.

One more thing follows from this and it reaches back to the published study. The item set retuning lowered the OCR rate from 0.028 to 0.01, so the published set carries [the same defect](https://dinand.com/research/studies-the-gates-stopped/#ocr-correction) at nearly three times the density. Rebuilding it at seed 90210 gives 63 of 150 items, 42%: invoice_id 25, po 14, subtotal 12, total 11, tax 9. I confirmed that reconstruction three ways. It hashes to `6db1ef80b3a1d143…`, the value the pre-registration pins for the published set. The broken juror scores 14 of 150 on it, which is the number the published article reported. Tax is zero on 25 of its items, which is the number quoted above for the same set. What that does to the published result is in the last section, and the short version is that it makes the published gap bigger.

## The roster, and which models the gate let through

Seven models ran, one loaded at a time, on two hosts. Base architecture was verified through the serving API before scoring and logged in the ledger: qwen2 for the 32B distill, llama for the 70B distill and both Llama 8B rungs, qwen3 and qwen3moe for the two Qwen3 models, gemma3 for the last. The pre-registration set a marginal error band of 10% to 90% and declared, before any full-run data existed, that a model outside it would be dropped from phi and from the difficulty vector alike.

Figure 1 · marginal error rate against the pre-registered band

*Every model's share of the 150 items answered wrong, recomputed from the per-call ledger. The band was fixed in the pre-registration at 10% to 90%. One model sits at 98.7% and is dropped from phi and from the difficulty vector, which the amendment declared before any full-run data existed. Both treatment models clear the band, so the primary contrast is defined.*

Six models cleared the band. The 8B distill did not, at 148 errors out of 150. Its floor is a documented quirk: on roughly half the items at every difficulty it returns the entire `INVOICE INV-…` header line as the invoice id, which the frozen exact-match rule scores wrong. It stays in the ledger and in the secondary cells, and it is excluded from the primary analysis exactly as the amendment said it would be. Both members of the treatment pair cleared the band, so the contrast is defined and the launch gate that armed the run held.

| model | base weights | fine-tune corpus | wrong | rate | gate |
| Qwen3 30B MoE (A3B) | Qwen3 30B MoE | none shared | 26/150 | 17.3% | pass |
| R1 distill 70B (treatment) | Llama 3.3 70B | R1 traces | 38/150 | 25.3% | pass |
| Gemma 3 27B | Gemma 3 | none shared | 39/150 | 26.0% | pass |
| R1 distill 32B (treatment) | Qwen2.5 32B | R1 traces | 59/150 | 39.3% | pass |
| Qwen3 8B | Qwen3 8B | none shared | 59/150 | 39.3% | pass |
| Llama 3.1 8B | Llama 3.1 8B | none shared | 119/150 | 79.3% | pass |
| R1 distill 8B | Llama 3.1 8B | R1 traces | 148/150 | 98.7% | fail |

Marginal error rates, recomputed from the ledger. The 30B MoE model answered in the pre-registered strip dialect after it declined the no-thinking flag; every other model answered with thinking disabled. Two items hit the token cap mid-reasoning and were coded as truncations.

## Conditioning on difficulty, exactly as before

The conditioning rule is the frozen one. Drop items where every model in the vector was right and items where every model was wrong, keep what is contested, recompute phi on those. The jurors that are not models stay out of the difficulty vector, as in the original. One thing changed and it was declared in advance: Amendment 2 restricts the difficulty vector to models that cleared the band, because a juror wrong 98.7% of the time turns nearly every item the others agree on into a spurious contest and dilutes phi for every pair.

With six models in the vector, 107 of 150 items are contested. Including the out-of-band model pushes that to 133 and lifts every phi in the matrix, which is the effect the amendment predicted. The published run kept 47 of 150 with four models in its vector, so this study's net is wider by construction, and the pre-registration said so at the time.

Figure 2 · what difficulty conditioning removes, cell by cell

*Cell means over the six gate-passing models. Raw phi runs above +0.40 for all three cells, which is what item difficulty looks like when it is left in. Conditioning drops all three together and the treatment cell lands between the other two rather than above them.*

## The result

Conditioned phi for the treatment pair is +0.1490. The ten cross-family control pairs average +0.1218. The registered contrast is therefore +0.0272, with a 95% bootstrap interval from -0.1637 to +0.2057 over 10,000 resamples of the 107 contested items. The interval contains zero and the point estimate is well under the +0.15 refutation threshold, which is the pre-registered condition for calling H1 refuted.

Figure 3 · the pre-registered contrast and what it rules out

*Delta is the treatment pair's conditioned phi minus the mean over the ten cross-family control pairs, with a 10,000-resample bootstrap over the 107 contested items. The interval contains zero and stops short of the +0.15 refutation threshold. Its upper edge reaches 38% of the way to the gap that H1 pointed at.*

The interval is worth reading for what it excludes. Its upper edge sits at +0.206, and the gap H1 pointed at, running from the published cross-family baseline up to the same-weights value, is +0.549. The data therefore rule out anything above 38% of that effect. A small positive movement of a tenth or two stays inside the interval and this study cannot settle whether it is there. The claim the study was built to test asked for something far larger, and it does not survive.

Ranking every pair on the same items puts the treatment sixth of fifteen. Nine cross-family and same-family pairs sit below it and five sit above, two of them far above at +0.499 and +0.546. Under an exact test on that ranking the treatment pair is ordinary, at p = 0.40.

Now the same block of numbers with the 24 defective items taken out and the roster held. Conditioned phi for the treatment pair is +0.039 and the controls average +0.114, so the contrast is -0.074 with an interval from -0.268 to +0.113 over 100 contested items. The pair ranks tenth of fifteen. Re-applying the error gate to the cleaned pool instead of holding the roster takes phi(T) to -0.094 and the contrast to -0.167, interval -0.355 to +0.033. All three readings clear the registered refutation condition, and the two cleaned ones put the treatment pair below its own control mean.

The one number in the registered analysis that leaned toward H1 was the joint-failure count. Twelve items had both treatment models wrong where 8.8 would be expected if they were independent, and a permutation test on that ranked it at p = 0.096, which is the kind of number that keeps you awake. Five of those twelve are defective items. What is left is 7 against 6.3 expected, at p = 0.449.

Figure 4 · every pair on the same 107 items, ranked

*Conditioned phi for all fifteen pairs among the gate-passing models, sorted. The treatment pair sits sixth. Two cross-family pairs run far above it at +0.499 and +0.546, and the lowest three pairs in the matrix are also cross-family, which is the spread the earlier study's -0.102 mean never summarised well.*

One nuance survives and belongs in the record. Phi for the treatment pair is modestly above zero, and a permutation test that shuffles one member's item order puts it at p ≈ 0.10 against a null of true independence. The two distills do agree on their failures a little more often than chance would give: 12 items where both are wrong against 8.8 expected. The average cross-family pair in this run does the same thing to about the same degree, which is why the two cells sit on top of each other in every cut of the data below.

## Every way I tried to break it

Difficulty conditioning is the step with the most freedom in it, so it went first. Six conditioning rules, from no conditioning at all through to the tightest band that still leaves a usable item count, and one that puts all seven models back in the vector. The contrast never clears zero under any of them, and under the tighter bands it goes negative.

Figure 5 · the contrast under six conditioning rules

*The conditioning step carries the most analytic freedom, so it was varied deliberately. Delta stays at or below zero everywhere and every interval spans zero. Tightening the band past the frozen rule pushes the contrast negative.*

A degenerate seventh variant belongs in the record because it is the only cut in the whole study that would support H1. Restricting to items where exactly three of six models were wrong leaves 11 items, on which the contrast reads +0.391 with an interval clear of zero. The two-by-two table behind that number is 2, 8, 0 and 1. A bootstrap over 11 fixed items cannot widen past that table, so the interval is not a confidence statement about anything, and a permutation test on the same band returns p = 0.82, so it goes in the log rather than in the result.

Next, the control set. Dropping any single control model in turn moves the contrast between -0.177 and +0.123, and every interval spans zero. Adding the out-of-band model back moves it to +0.009. Re-seeding the bootstrap five times leaves the point estimate unchanged at +0.0272 and the interval edges stable inside a hundredth. Dropping the ten items where any gate-passing model returned something other than a clean answer or a clean error takes the contrast to +0.077, still inside a zero-spanning interval.

Both of those sweeps were re-run on the cleaned pool after the fact, and every cell moves down. The conditioning ladder reads -0.139 with no conditioning, -0.074 under the frozen rule, -0.138 on the 1-to-4 band, -0.274 on 2-to-5, -0.341 on the tightest usable band, and -0.093 with all seven models in the vector. Leave-one-out over the controls runs from -0.290 without Llama3.1-8B to +0.045 without Gemma3-27B. Nothing in either sweep crosses the refutation threshold and most of it sits well below zero.

Figure 6 · leave-one-out over the control set

*The headline contrast with each control model removed in turn, and with the out-of-band model added back. The range runs from -0.177 to +0.123 and no interval clears zero.*

## The secondary cells

Three other cells were pre-registered and all three are reported here. The pair that Amendment 1 demoted, the 32B distill against the 8B one, reads +0.060 against a control mean of +0.202, for a contrast of -0.142 with an interval from -0.188 to -0.067. Read literally that clears zero on the negative side, which would say the shared-corpus pair is *less* correlated than cross-family models, and reading it that way would be a mistake. The 8B distill's two-by-two against the 32B is 59, 0, 89 and 2, meaning it was wrong on all but two items and there is almost no variance for phi to work with. The pre-registration predicted this cell would come out variance-dead and it did.

The shared-base-weights cell, which puts the 8B distill against the Llama 3.1 model it was built from, reads -0.047. The cell that shares both the corpus and the Llama lineage reads +0.039. Both carry the same variance problem and neither should be quoted as an effect. The pre-registered ordering, which predicted shared weights at or above shared data and both above cross-family, cannot be evaluated on these numbers.

| cell | what it shares | pairs | raw phi | conditioned | raw, minus the 24 | conditioned, minus the 24 | status |
| T treatment | fine-tune corpus, no weights | 1 | +0.410 | +0.149 | +0.102 | +0.039 | primary |
| F family | lineage, no shared corpus | 4 | +0.425 | +0.184 | +0.230 | +0.161 | reported |
| X control | nothing | 10 | +0.426 | +0.122 | +0.241 | +0.114 | baseline |
| T2 secondary | fine-tune corpus, no weights | 1 | +0.094 | +0.060 | +0.082 | +0.058 | variance-dead |
| W secondary | pretrained base weights | 1 | +0.084 | −0.047 | +0.078 | −0.049 | variance-dead |
| TW secondary | corpus and base lineage | 1 | +0.068 | +0.039 | +0.050 | +0.036 | variance-dead |

Cell means. T, F and X are computed over the six gate-passing models on 107 contested items, and on 100 in the cleaned columns. The three secondary cells all contain the out-of-band model, so their conditioned values come from the seven-model vector on 133 items and 125 cleaned, and are not comparable to the rows above them. The secondary cells barely move, which is what a variance-dead cell does when you take items away from it.

## The independence floor I thought I had measured

Against raw error vectors the repaired programmatic juror shows positive phi with all six gate-passing models, from +0.048 up to +0.372, and it goes slightly negative only against the model that failed the band, at -0.073. After difficulty conditioning it spreads from -0.158 to +0.231, with the two strongest models at the top of that range.

I read that as a clean measurement of a non-model independence floor, and wrote that a working floor sits nearer +0.1 than 0 on raw scores. That claim is withdrawn. The juror shares nothing with a language model except the documents, and once the 24 defective documents come out, the correlation goes with them.

| phi between the repaired juror and each model | all 150 items | minus the 24 |
| raw, across the six gate-passing models | +0.048 to +0.372 | −0.064 to +0.104 |
| after difficulty conditioning | −0.158 to +0.231 | −0.200 to +0.108 |
| juror score | 115 of 150 | 108 of 126 |

Mean raw phi across the six models falls from +0.222 to +0.007. A regular expression measured against a language model on documents both can read sits on zero, which is what it should have done all along.

So the honest version is narrower than what I had. On a clean item set I have no evidence of a non-model independence floor above zero, and no evidence against one either, because the effect I was reading turned out to be the defect. The published claim that a non-model juror sits near zero against everything came from a juror reading the wrong field, so it was right for the wrong reason. What a juror that inspects fewer fields costs you is still an open question and this run cannot answer it.

Figure 7 · the programmatic juror, before and after repair

*The non-model juror included as an independence floor. The published version searched its labels case-insensitively and matched inside "Subtotal", so it compared subtotal plus tax against the subtotal. Model scoring runs through a different code path and is unaffected, so no published model-pair number moves.*

## Limits

- The treatment is one pair. There are exactly two capability-matched distills of the same corpus onto unrelated bases available to run, so the cell holds a single number and the bootstrap interval on it is wide.
- Whether the pretraining corpora of the control models ingested R1-generated text is unknowable from outside. The controls are classified as cross-family on lineage, and if any of them quietly absorbed reasoning traces then the baseline is contaminated in the direction that would *help* H1. The hypothesis failed against that baseline regardless.
- The item set differs from the published one, so the phi levels here are not directly comparable to +0.447 and -0.102. The within-run contrast carries the result.
- Sixteen percent of the items ask for a string the document does not contain. Every headline number appears twice in this article for that reason. The registered analysis is the one I pre-committed to and I report it first; the cleaned reading is the one I would design for if I ran this again. They agree on the verdict and differ on how firmly.
- The cleaned reading has a roster problem with no clean answer. Removing the defective items drops the Qwen3 MoE below the pre-registered error floor, so either the gate moves or the roster does. I report both and neither changes the verdict.
- Nothing here separates a model that reads `INV-1O0114` literally from one that fails the item for an unrelated reason. The ledger records pass and fail per item and not per field, so the mechanism behind each failure on those 24 items is inferred from the item and not observed in the output. Re-running with per-field detail would settle it.
- One extraction task, ground truth by construction. Correlated failure on open-ended reasoning could behave differently and this says nothing about it.
- The Qwen3 family documents strong-to-weak distillation inside itself, so the F cell is not a clean no-shared-signal comparison. It is reported as context rather than as a control.

## What this changes about the earlier article

The arithmetic of that article is intact and I checked it rather than assuming so. The same-weights and cross-family numbers recompute to +0.4475 and -0.1015 on 47 contested items from the archived report, off bit-identical vectors. What has changed since I wrote that sentence is what those items were, and that is the subject of two paragraphs further down.

The sentence about the deterministic checker does not survive and needs a correction note. Its juror was reading the subtotal as the total on 147 of 150 items and its near-zero phi is the signature of an answer chosen close to at random. The correction is that the published number was right by accident, and it stops there. Repairing the juror does not license a claim about a positive independence floor, because on documents that both sides can read the repaired juror sits on zero too.

The bigger correction is the item set, and it runs the other way from what I expected. The published run used the hard-mode generator at an OCR rate of 0.028, nearly three times this study's, and 63 of its 150 items, 42%, ask for a string the document does not contain: invoice_id 25, po 14, subtotal 12, total 11, tax 9. Eleven more carry an amount with a comma for a decimal point. I rebuilt that set from the archived harness at seed 90210, confirmed it hashes to the pinned value, and re-scored the archived error vectors with the defective items dropped.

| published study, conditioned phi | as published | minus the 63 defective items |
| same weights, different silicon | +0.4475 | +0.5205 |
| cross-family mean | −0.1015 | −0.0946 |
| the gap | +0.5490 | +0.6151 |
| contested items | 47 of 150 | 36 of 87 |

Four models, the frozen conditioning rule, the archived error vectors untouched. Forty-three of the 49 items that every model failed were defective ones, so the conditioning rule was again absorbing most of the damage before it reached the estimate.

So the published headline holds and gets larger. The defect was pulling that gap down, because an item nobody can answer pushes the cross-family cell up alongside the same-weights cell and narrows the distance between them. The cross-family baseline barely moves at all, which is the part I would have bet against.

The published juror is the one place study 1 gets off lightly for the wrong reason. It was wrong on 91% of everything, so the defective items cost it almost nothing: 92.1% error on those against 89.7% on the rest. Its phi against the models was negative before and stays negative after, from -0.213 to -0.095. The correction it needs is still the regular expression, and only that.

The open question that article named now has an answer, and the answer is no. Sharing a fine-tune corpus, with nothing shared in the weights, leaves error correlation sitting at the cross-family level. Whatever running identical weights does to two systems, a shared training signal on its own does not reproduce it.

Last, the cross-family baseline turns out to be a property of the item set and the conditioning net rather than a constant of nature. It reads -0.102 on the published items with four models in the vector and +0.122 here with six. Anyone quoting either number needs to carry the item set with it.

## Reproduction

The run wrote 1,050 per-call records to the raw call ledger, one per model per item, each carrying the item-set hash, the serving dialect, the outcome code and the latency. Verification rebuilt all seven error vectors from those records with an independent parser and compared them to the run's own report: seven of seven match exactly. The item set was regenerated from the seed and hashed. Both jurors were reimplemented from scratch and reproduce the run's vectors bit for bit. The contrast, the bootstrap, the cell means and every sensitivity line above were computed by that independent code, and where the run's own analysis reports the same quantity the two agree to three decimals on every pair.

The wall clock reads 10 hours 51 minutes, of which 5 hours 15 minutes was the 70B pass waiting for an unrelated job to release the second host, under a mechanical gate written into the harness after an earlier pilot evicted that job's model once. 168 of the 1,050 records were reused from the pilot ledger under the same item hash and the same temperature, and each reuse is logged.

---
Site guide for AI agents: https://dinand.com/llms.txt