Measured 150 items, four model configurations
The same model on different silicon shares its errors (phi +0.447 on contested items); models from different families do not (mean phi −0.102).
Registration and verificationGround truth is generated with each item, so no model grades another.
A common resilience design runs the same model a second time, on different accelerators or in a second region, and treats the second answer as a check on the first. Civil aviation learned long ago that two identical computers fail the same way at the same moment. This study asked whether a second copy of a model, on different hardware, makes different mistakes.
Ground truth is generated with each item, so the correct answer is known before any model sees it and no model grades another.
Four model configurations scored the same 150 invoice-extraction items. The key comparison ran qwen3.8 twice, once as NVFP4 on SGLang on an NVIDIA node and once as Q4_K_M on ollama on an AMD card: identical weights, different quantisation, silicon and serving stack. For each pair the study computed the phi coefficient on error indicators, near zero when two models go wrong on different items and near one when they go wrong together.
103 of the 150 items were unanimous, 54 that every model got right and 49 that every model got wrong. Those say how hard an item is and nothing about shared weakness, so the headline uses the 47 items where the models disagreed.
Before conditioning, every pair looked correlated, from 0.556 to 0.827. On the contested items, the pair with identical weights on different hardware stayed at +0.447, the highest in the matrix. The five pairs from different model families averaged −0.102, and qwen3.8 NVFP4 against gemma3 sat at −0.469; gemma3 against qwen3.6 was −0.105. Changing the hardware, the quantisation and the serving stack left the correlation where it was. Changing the model family moved it through zero.
| Error correlation (phi), identical weights on different silicon, contested items only | +0.447 |
|---|---|
| Mean error correlation (phi) across five cross-family pairs, contested items only (range −0.469 to +0.280) | −0.102 |
| Error correlation (phi) across all pairs before conditioning on item difficulty | 0.556 to 0.827 |
| Pair | All 150 items | 47 contested items |
|---|---|---|
| Same weights, two machines | 0.827 | 0.447 |
| qwen3.8 Q4 + qwen3.6 | 0.810 | 0.280 |
| qwen3.8 NVFP4 + qwen3.6 | 0.729 | 0.013 |
| gemma3 + qwen3.6 | 0.617 | −0.105 |
| qwen3.8 Q4 + gemma3 | 0.607 | −0.227 |
| qwen3.8 NVFP4 + gemma3 | 0.556 | −0.469 |
“Better questions lead to better worlds.”Dinand Tinholt