# The Second Opinion Still Holds Source: https://dinand.com/research/second-opinion-still-holds/ Author: Dinand Tinholt Published: 2026-09-08 > Buying a second opinion from a different model family still works: a shared training corpus did not make two models share blind spots. The Second Opinion Still Holds Executive summary · error independence, second study # Buying a second opinion from a different model still works The worry was reasonable. If half the field fine-tunes on the same reasoning data, then picking two different models for a review step might buy you two copies of the same blind spot without you noticing. We tested it directly, on two models that share a fine-tune corpus and share nothing in their weights. They fail together no more often than any unrelated pair of models does, so a review step built on model diversity is still doing what it was bought to do. A defect in the test documents turned up after the run, affecting 16% of them, and every number here is given twice because of it. The answer is the same under both. Pre-registered before any measurement, refutation published as registered · 7 models, 150 items, 1,050 scored calls · every figure recomputed from the raw call log by an independent check +0.149 how much the shared-corpus pair fails together +0.122 how much unrelated pairs fail together, same items +0.027 the difference, with zero inside its error bar +0.447 what identical weights on different hardware look like 6 of 15 where the shared-corpus pair ranks among all pairs ## The question A lot of controls in production systems assume two reviewers make different mistakes. Maker and checker, or a second model verifying the first. Either one is worth the extra spend only when the second opinion is genuinely a second opinion. Our earlier work measured that assumption and found it breaks in one specific way: run the same model weights on two different machines with two different quantisations and you get the same errors on the same items, at a correlation of +0.447. Pick models from different families and the correlation drops to -0.102, which is what a useful second opinion looks like. That left an uncomfortable gap. Model families are converging on the same training material. Reasoning traces generated by one lab's model get used to fine-tune other labs' models, and that happens across weight lineages that have nothing to do with each other. If a shared training corpus is enough to install a shared blind spot, then a requirement to "use a different model family" would be doing nothing at all while looking like a control. ## How we tested it One vendor published a set of models that make the test clean. They took a single collection of roughly 800,000 reasoning examples and fine-tuned it onto base models from completely unrelated pretraining runs. Two of those models therefore share every example in the fine-tune and share nothing whatsoever in their weights. That is the treatment. Against it we ran five control models that share no fine-tune corpus with anything else in the set, giving ten unrelated pairs measured on exactly the same items by exactly the same harness. The task is invoice extraction where the answer key is generated along with the document, so no model ever grades another and there is no judge to argue with. Each of the seven models answered all 150 items. That is 1,050 scored calls, all logged. Then the analysis conditions on difficulty. Items that everybody got right and items that everybody got wrong tell you nothing about whether two models fail *together*, so they come out and the correlation is recomputed on the 107 items that were genuinely contested. This step is the one that overturned the raw numbers in the earlier study, and it is applied here unchanged. ## The answer The shared-corpus pair correlates at +0.149. The ten unrelated pairs average +0.122. The difference is +0.027 and its 95% interval runs from -0.164 to +0.206, so zero sits comfortably inside it. Under the rule we registered before collecting any data, that reads as a refutation. Figure 1 · where shared training data actually lands *Each point is the correlation between two models' errors on the items they genuinely contested. The right-hand point is the pair that shares a fine-tune corpus and no weights, with the 95% interval on its gap to the controls beside it. The two left-hand points come from the earlier study on a harder document set and mark the scale rather than sitting on the same axis of comparison.* Ranking all fifteen pairs on the same items puts the shared-corpus pair sixth. Five pairs correlate more strongly and nine correlate less, and two of the pairs above it are ordinary cross-family combinations sitting at +0.499 and +0.546. On this evidence a shared fine-tune corpus does not mark a pair out as special. | what the pair shares | pairs | correlation | reading | | --- | --- | --- | --- | | Identical weights, different hardware (earlier study) | 1 | +0.447 | the second opinion adds little | | Same fine-tune corpus, unrelated weights | 1 | +0.149 | indistinguishable from unrelated | | Same family lineage, no shared corpus | 4 | +0.184 | indistinguishable from unrelated | | Nothing shared | 10 | +0.122 | the baseline for this run | | Nothing shared (earlier study, harder items) | 5 | -0.102 | the baseline for that run | Correlation on contested items. The two rows marked as the earlier study were measured on a different, harder set of documents, so they indicate the scale rather than sitting directly alongside the rows above them. The comparison this study actually tests is between rows two and four, which were scored on identical items by identical code. ## How hard we tried to break it A single number with a wide error bar is easy to talk yourself into. So the result was attacked from several directions before we accepted it. Removing any one control model from the comparison and recomputing leaves the difference between -0.177 and +0.123, and every one of those intervals contains zero. Putting back a model that had failed a pre-registered quality gate moves the difference to +0.009. Varying the difficulty-conditioning rule across six settings, from no conditioning at all to the tightest band that still leaves enough items to work with, never produces a positive result that clears zero. Re-seeding the resampling five times moves nothing. Figure 2 · the same result, cut six different ways *The gap between the shared-corpus pair and the unrelated controls, recomputed with each control model removed in turn and with a model that failed the quality gate added back. The range runs from -0.177 to +0.123 and no interval clears zero.* One cut of the data does support the original hypothesis, and it is worth saying so out loud. Restricting to items where exactly three of six models were wrong leaves eleven items and produces a difference of +0.391. The contingency table behind that number has a single observation in one of its cells, and a permutation test on the same eleven items returns p = 0.82, which puts it in the log rather than in the finding. ## What this means for review and jury design The control that works is choosing a genuinely different model. That conclusion is now supported against two separate threats rather than one. Running the same weights on different infrastructure buys nothing, which the earlier study established. Two models sharing a fine-tune corpus behave like any other unrelated pair, which is what this study adds. The thing that made correlated failure appear in the first study was the weights themselves. Two practical readings follow. Diversity requirements written into a control can be stated over model lineage without also needing a clause about training provenance, which is fortunate, because training provenance is usually unknowable from outside anyway. And redundancy that varies the serving stack or the hardware while keeping the weights should stop being counted as independence, since it measures at +0.447 as published and +0.521 once the defective documents come out of that study too, and a second opinion at either correlation is close to the first opinion repeated. What this does not say. The interval on our result reaches +0.206 at its upper edge, so a small effect from shared training data cannot be ruled out. What is ruled out is anything approaching the size the hypothesis predicted, which was roughly +0.55. The measurement covers one extraction task with a machine-generated answer key. Correlated failure on open-ended reasoning work could behave differently and nothing here speaks to it. ## A defect in the test documents, found late The documents are generated with scanner-style noise applied on top, which swaps a 0 for an O, a 1 for an l, a 5 for an S, an 8 for a B. That noise runs over the whole page. It reaches inside the invoice number and inside the amounts, which are the fields the scoring code then checks by exact match. So a document can read `INV-1O0114` while the answer key insists on `INV-100114`, and every model gets marked wrong on a field it read correctly. This affects 24 of the 150 documents, 16%. We found it during the writing, which is late. It is in here because it moves numbers you would otherwise quote. Every headline in this summary was recomputed with those 24 documents removed. The answer holds and gets firmer. | | as run | with the 24 removed | | shared-corpus pair | +0.149 | +0.039 | | unrelated pairs, average | +0.122 | +0.114 | | the difference | +0.027 | −0.074 | | 95% interval on the difference | −0.164 to +0.206 | −0.268 to +0.113 | | where the shared-corpus pair ranks | 6 of 15 | 10 of 15 | The defect had been pushing the shared-corpus pair upward, so cleaning it moves the result further from the hypothesis. A stricter reading that also re-applies the study's model quality gate to the smaller document pool puts the difference at −0.167. One thing the defect does break. We had read the repaired non-model checker as evidence that even a plain piece of code correlates a little with the models. That correlation was the broken documents. Take them out and it falls from a range of +0.05 to +0.37 down to −0.06 to +0.10, which is zero. A regular expression shares nothing with a 70B model, and on documents both can read it behaves that way. The claim about a floor is withdrawn. ## A correction to the earlier work Verification of this run turned up a defect in the published harness, and it belongs in the open. The earlier study included a small non-model checker, a piece of ordinary code that recomputes a few fields straight from the document, as a reference point for what real independence looks like. Its pattern matching was case-insensitive, so its search for "TOTAL" matched inside the word "Subtotal" on 147 of the 150 documents and it compared the wrong pair of numbers. It scored 14 out of 150 and showed near-zero correlation with every model, which is what an answer close to random looks like. Model scoring runs through separate code and never touched the defect, so every model-to-model number in the published article is unaffected. We confirmed that by recomputing the published headline from the archived per-item results: +0.4475 and -0.1015, matching what was published. The sentence that needs correcting is the one describing the non-model checker as an independence floor. It was near zero because it was reading the wrong field, so the number was right by accident. Repairing it does not put a floor above zero either, for the reason set out two sections up. The [document defect](https://dinand.com/research/studies-the-gates-stopped/#ocr-correction) reaches back into that study as well, and harder. The published run used the same generator at nearly three times the noise rate, so 63 of its 150 documents, 42%, ask for a string the page does not carry. We rebuilt that document set from its seed, confirmed it against the recorded hash, and re-scored the archived results with those 63 dropped. | published study | as published | with the 63 removed | | same weights, different hardware | +0.4475 | +0.5205 | | unrelated pairs, average | −0.1015 | −0.0946 | | the gap | +0.5490 | +0.6151 | Four models, the same conditioning rule, the archived per-item results untouched. That headline holds and grows. A document nobody can read correctly lifts every pair at once, so it was closing the gap between the same-weights pair and the unrelated ones rather than opening it. The published article needs a note about its document set. Its conclusion needs nothing. ## Confidence in the numbers Everything above was recomputed from the raw per-call log by a separate parser written for the check, without reusing the run's own analysis code. That check is also what turned up the document defect, and it did so after the draft was written rather than before, which is the honest order of events. All seven models' results rebuilt from that log match the run's report exactly. The document set was regenerated from its seed and hashed against what the pre-registration recorded, and both versions of the checker were reimplemented from scratch and reproduce the run's numbers to the item. Where the run's own analysis reports the same quantity, the two agree to three decimals on every pair. The hypothesis, the direction, the comparison and the threshold for calling it refuted were all written down before any measurement was taken, along with a commitment to publish a refutation. Two amendments were made after a pilot run and before any measurement, one replacing a treatment model that was too weak for the task to produce usable variance, and one retuning the documents so that models were not all pinned at their error ceiling. Both amendments changed the document set away from the published one, which is why the two studies' absolute numbers are quoted here as scale rather than as a direct comparison. --- Site guide for AI agents: https://dinand.com/llms.txt