Concepts

Error Independence

The property that makes a second opinion worth having: when one checker is wrong, the other is wrong on different items.

What it means

A common resilience design runs the same model a second time, on different hardware or in a second region, and treats the second answer as a check on the first. Error independence is the property that makes a second opinion worth having: when one checker is wrong, the other is wrong on different items.

Civil aviation learned long ago that two identical computers fail the same way at the same moment. The lab asked whether the same holds for models. Four configurations scored the same 150 invoice-extraction items, and for each pair the study computed the phi coefficient on their errors, near zero when two models go wrong on different items and near one when they go wrong together.

Before conditioning on difficulty, every pair looked correlated, from 0.556 to 0.827. On the 47 contested items, the pair with identical weights on different silicon and a different serving stack stayed at phi +0.447, the highest in the matrix. The five pairs from different model families averaged phi -0.102.

Changing the hardware left the correlation where it was. Changing the model family moved it through zero. A second opinion that shares the blind spot confirms the first one's mistakes. The same-family reading comes from a single pair, and the task was one where ground truth cannot be argued with.

Headline number

Error correlation (phi) between identical weights on different hardware, contested items; the mean across cross-family pairs is −0.102+0.447

Where it comes from

Related concepts

  • Decision HeadA small service that takes the state of a case and a list of typed questions and returns a probability for each answer, on a model you host.
  • Delegation CliffThe point where pushing work further from human review stops paying for itself.
“Better questions lead to better worlds.”Dinand Tinholt

Follow your curiosity.

Surprise me
Top