# Error Independence: does different hardware buy a second opinion?

Source: https://dinand.com/research/campaigns/error-independence/
Author: Dinand Tinholt

> The same model on different silicon shares its errors (phi +0.447 on contested items); models from different families do not (mean phi −0.102).

## The question

A common resilience design runs the same model a second time, on different accelerators or in a second region, and treats the second answer as a check on the first. Civil aviation learned long ago that two identical computers fail the same way at the same moment. This study asked whether a second copy of a model, on different hardware, makes different mistakes.

## What was registered

Ground truth is generated with each item, so the correct answer is known before any model sees it and no model grades another.

## What was measured

Four model configurations scored the same 150 invoice-extraction items. The key comparison ran qwen3.8 twice, once as NVFP4 on SGLang on an NVIDIA node and once as Q4_K_M on ollama on an AMD card: identical weights, different quantisation, silicon and serving stack. For each pair the study computed the phi coefficient on error indicators, near zero when two models go wrong on different items and near one when they go wrong together.

103 of the 150 items were unanimous, 54 that every model got right and 49 that every model got wrong. Those say how hard an item is and nothing about shared weakness, so the headline uses the 47 items where the models disagreed.

## What it found

Before conditioning, every pair looked correlated, from 0.556 to 0.827. On the contested items, the pair with identical weights on different hardware stayed at +0.447, the highest in the matrix. The five pairs from different model families averaged −0.102, and qwen3.8 NVFP4 against gemma3 sat at −0.469; gemma3 against qwen3.6 was −0.105. Changing the hardware, the quantisation and the serving stack left the correlation where it was. Changing the model family moved it through zero.

Headline numbers, as published
| Error correlation (phi), identical weights on different silicon, contested items only | +0.447 |
| --- | --- |
| Mean error correlation (phi) across five cross-family pairs, contested items only (range −0.469 to +0.280) | −0.102 |
| Error correlation (phi) across all pairs before conditioning on item difficulty | 0.556 to 0.827 |

*Pairwise error correlation (phi) on all 150 items (open ring) and on the 47 contested items (filled dot). The highlighted pair runs identical weights on different hardware. Rebuilt from the numbers in [When the Second Opinion Shares the Blind Spot](https://medium.com/@tinholt/when-the-second-opinion-shares-the-blind-spot-7e4443dcb081).*

Show the numbers as a table

| Pair | All 150 items | 47 contested items |
| Same weights, two machines | 0.827 | 0.447 |
| qwen3.8 Q4 + qwen3.6 | 0.810 | 0.280 |
| qwen3.8 NVFP4 + qwen3.6 | 0.729 | 0.013 |
| gemma3 + qwen3.6 | 0.617 | −0.105 |
| qwen3.8 Q4 + gemma3 | 0.607 | −0.227 |
| qwen3.8 NVFP4 + gemma3 | 0.556 | −0.469 |

## What it does not claim

- The same-family reading comes from one pair, the only two systems in the set with identical weights. One pair is an observation.
- The five cross-family pairs run from −0.469 to +0.280, and the mean sits inside a wide spread without describing it well.
- One task, chosen because its ground truth cannot be argued with. Open-ended work is untested.

## Follow-up: does shared training data correlate errors?

If half the field fine-tunes on the same reasoning data, two different models might share a blind spot anyway. A pre-registered second study tested that on seven models, 150 items and 1,050 scored calls. The pair that shares a training corpus correlated at +0.149 against +0.122 for unrelated pairs, a difference of +0.027 (95% interval −0.164 to +0.206). The hypothesis was refuted: models from different families do not share blind spots, and a shared training corpus did not correlate their errors.

Follow-up and corrected numbers
| Shared-corpus pair minus unrelated pairs, error correlation (95% interval −0.164 to +0.206) | +0.027 |
| Identical weights on different silicon, with the 63 defective documents removed (published +0.447) | +0.521 |
| Documents affected by the generator defect | 63 of 150 (42%) |

## Read the pieces

- [When the Second Opinion Shares the Blind Spot](https://medium.com/@tinholt/when-the-second-opinion-shares-the-blind-spot-7e4443dcb081)Paper, Medium, 2026-09-02
- [Shared Training Data Does Not Correlate Errors](https://dinand.com/research/shared-training-data/)Follow-up paper, 2026-09-08
- [The Second Opinion Still Holds](https://dinand.com/research/second-opinion-still-holds/)Executive summary of the follow-up, September 2026
- [A Ceiling on Herding in a Coupled Model Fleet](https://dinand.com/research/herding-ceiling/)Related campaign: [Herding](https://dinand.com/research/campaigns/herding/)

---
Site guide for AI agents: https://dinand.com/llms.txt