When you say three models "agree" about a brand, what do you actually mean? Two of them said the founding year was 2019 and one said 2018 — is that agreement? Two said "$49/mo" and one refused to answer — is that agreement? You need a number, and the obvious number is wrong.
Why raw agreement lies
The naive metric is percent agreement: count the questions where all three models said the same thing, divide by total. It's intuitive and it's misleading, because it ignores chance. On a question with only two plausible answers, two models will agree by luck half the time. Percent agreement rewards easy questions and punishes hard ones, which is exactly backwards.
Cohen's kappa fixes this by subtracting out expected-by-chance agreement:
κ = (p_observed − p_chance) / (1 − p_chance)
A κ of 1.0 means perfect agreement. A κ of 0 means the models agree exactly as often as random guessing would predict. Negative κ means they systematically disagree.
The adaptation problem
Cohen built kappa for two human raters scoring a fixed rubric. We have three stochastic models, each run multiple times, producing free-text answers that have to be bucketed into verdicts first. Three things break:
- Three raters, not two. We move to Fleiss' kappa, which generalizes to any number of raters.
- Stochastic raters. The same model gives slightly different answers across runs. We treat each model-question pair as a single rater by majority-voting its three runs, then propagate the within-model variance as a confidence interval.
- Verdict bucketing. Free text has to become a category — correct, incorrect, refused — before kappa applies. That bucketing step is where most of the noise lives, so we double-rate a sample by hand to calibrate it.
What the numbers came out to
Across our sixty-question, three-model panel, inter-model κ landed at 0.62. In the human-rater literature that's "substantial agreement" — but for our purposes the absolute number matters less than the per-question spread.
The questions with κ near 1.0 are boring: everyone agrees, nothing to fix. The questions with κ near 0 are the product. Those are the brand facts where the models genuinely diverge, which is exactly where a brand should spend its AEO effort.
"We don't report the average κ to customers. We report the lowest κ questions, because those are the ones costing you accuracy."
The honest caveat
Kappa is a tool built for a world of stable, discrete-choice raters. LLMs are neither stable nor discrete. Every adaptation we made introduced an assumption, and we document each one in the open-source methodology repo. If you think we got the bucketing wrong, the code is there to prove it. That's the point of publishing the math — so the disagreement about how we measure disagreement can itself be measured.