Measured 150 items, four model configurations
The same model on different silicon shares its errors (phi +0.447 on contested items); models from different families do not (mean phi −0.102).
Registration and verificationGround truth is generated with each item, so no model grades another.
Two follow-up studies found two faults behind the published piece. The headline holds, and grows.
The deterministic checker, included as a non-model juror and an independence floor, searched for its labels case-insensitively, so the pattern for TOTAL matched inside “Subtotal:” and grabbed the wrong number. It was near zero because it was reading the wrong field. The sentence about an independence floor is withdrawn.
The document generator ran at nearly three times the intended noise rate, so 63 of the 150 documents, 42%, ask for a string the page does not carry. With those removed, the same-weights pair measures +0.521 against +0.447 as published. The shared correction note covers the defect.
The second study also withdraws its own earlier reading that hand inspection showed the failures were real rather than a harness artifact. The models were transcribing what the page said. Details under the follow-up.
A common resilience design runs the same model a second time, on different accelerators or in a second region, and treats the second answer as a check on the first. Civil aviation learned long ago that two identical computers fail the same way at the same moment. This study asked whether a second copy of a model, on different hardware, makes different mistakes.
Ground truth is generated with each item, so the correct answer is known before any model sees it and no model grades another.
Four model configurations scored the same 150 invoice-extraction items. The key comparison ran qwen3.8 twice, once as NVFP4 on SGLang on an NVIDIA node and once as Q4_K_M on ollama on an AMD card: identical weights, different quantisation, silicon and serving stack. For each pair the study computed the phi coefficient on error indicators, near zero when two models go wrong on different items and near one when they go wrong together.
103 of the 150 items were unanimous, 54 that every model got right and 49 that every model got wrong. Those say how hard an item is and nothing about shared weakness, so the headline uses the 47 items where the models disagreed.
Before conditioning, every pair looked correlated, from 0.556 to 0.827. On the contested items, the pair with identical weights on different hardware stayed at +0.447, the highest in the matrix. The five pairs from different model families averaged −0.102, and qwen3.8 NVFP4 against gemma3 sat at −0.469; gemma3 against qwen3.6 was −0.105. Changing the hardware, the quantisation and the serving stack left the correlation where it was. Changing the model family moved it through zero.
| Error correlation (phi), identical weights on different silicon, contested items only | +0.447 |
|---|---|
| Mean error correlation (phi) across five cross-family pairs, contested items only (range −0.469 to +0.280) | −0.102 |
| Error correlation (phi) across all pairs before conditioning on item difficulty | 0.556 to 0.827 |
| Pair | All 150 items | 47 contested items |
|---|---|---|
| Same weights, two machines | 0.827 | 0.447 |
| qwen3.8 Q4 + qwen3.6 | 0.810 | 0.280 |
| qwen3.8 NVFP4 + qwen3.6 | 0.729 | 0.013 |
| gemma3 + qwen3.6 | 0.617 | −0.105 |
| qwen3.8 Q4 + gemma3 | 0.607 | −0.227 |
| qwen3.8 NVFP4 + gemma3 | 0.556 | −0.469 |
If half the field fine-tunes on the same reasoning data, two different models might share a blind spot anyway. A pre-registered second study tested that on seven models, 150 items and 1,050 scored calls. The pair that shares a training corpus correlated at +0.149 against +0.122 for unrelated pairs, a difference of +0.027 (95% interval −0.164 to +0.206). The hypothesis was refuted: models from different families do not share blind spots, and a shared training corpus did not correlate their errors.
| Shared-corpus pair minus unrelated pairs, error correlation (95% interval −0.164 to +0.206) | +0.027 |
|---|---|
| Identical weights on different silicon, with the 63 defective documents removed (published +0.447) | +0.521 |
| Documents affected by the generator defect | 63 of 150 (42%) |
“Better questions lead to better worlds.”Dinand Tinholt