This is the deep cut from our model-selection study. The headline finding — three models capture 94% of the disagreement signal — got the attention. The omission is the more interesting story: the best raw reasoner in our test didn't make the panel.
The case for Claude
Claude 3.5 Sonnet scored highest on factual accuracy when handed good context. On pure reasoning quality it was, by most of our measures, the strongest of the eleven models we tested. If we were building a tool to answer brand questions, it would be an obvious pick.
But we're not building that. We're building a tool to measure disagreement. And for that job, raw quality is the wrong axis.
Why it didn't make the cut
The disqualifying number: Claude agrees with ChatGPT 81% of the time on brand questions. The two models share enough pretraining data and human-feedback signal that they make similar mistakes and reach similar correct answers. They're correlated raters.
For consensus measurement, two correlated raters are nearly one rater. Adding Claude to a panel that already has ChatGPT buys you very little new signal — about two percentage points of variance in our test, at full marginal cost. Perplexity, which agrees with everyone far less, buys you twenty-three.
"Diversity of method matters more than quality of method when you're sampling for consensus. The best third model is the one that disagrees most, not the one that's smartest."
What we do with Claude instead
Claude isn't gone — it's the watchdog. We run it monthly as a sanity check against the primary panel. If Claude suddenly starts agreeing with our three on a question they used to dispute, that's a signal the whole ecosystem has converged, and we should look at why.
We'll revisit the decision as Claude's web-search and grounding evolve. Today its grounding tracks ChatGPT's closely; the day that changes, its disagreement value goes up and it earns a second look. Until then, diversity wins.