Calibration: how probability forecasts are checked against outcomes

How the check works

Three steps.

1. 1
   Record the forecasts, dated, before the outcomes are known

   A probability reconstructed after the outcome is known can still be compared to it arithmetically, but the comparison no longer measures forecasting. Recollection tends to drift toward what happened.

2. 2
   Bucket them by stated probability

   The 50s, the 60s, the 70s, the 80s. Ten-point buckets rather than finer ones, because a bucket needs a reasonable number of resolved forecasts in it before its hit rate means much, and narrower bands spread the same forecasts thinner.

3. 3
   Compare each bucket's hit rate to its label

   The 70–79% bucket should resolve yes roughly 70–79% of the time.

Where the hit rate tracks the label, the forecasts are calibrated in that range. Where it doesn't, the gap says which range is off and by how much.

Worked example: a 60-forecast log

The log below is illustrative: sixty invented forecasts with invented outcomes, spanning elections, economic prints, climate records and crypto thresholds. It is grouped by the probability each forecast stated, not by subject, because calibration compares stated probabilities to outcomes and the subject doesn't enter that comparison.

The two lower buckets track their labels. The two upper ones don't: forecasts stated at 70% resolved yes closer to 57% of the time, and forecasts in the 80s resolved yes around 67% of the time.

What a table like this adds over a single score is location: it points at a range rather than at the forecaster generally. In this invented example the gap sits in the two high-confidence buckets and not in the lower two, which is a narrower finding than "be less confident."

It doesn't follow that any particular adjustment is the right response. One thing a forecaster could do with a result like this is state lower numbers in the range where the gap appeared and re-check on the next batch of resolved forecasts. But that's an illustration of how the diagnostic gets used, not a recommendation, and with buckets this small the gap could also be sampling noise rather than a habit. The next section is about telling those apart.

Reading a calibration curve

Plotting stated probability on the x-axis against observed hit rate on the y-axis, one point per bucket, gives the standard reliability chart. A perfectly calibrated set of forecasts sits on the 45° diagonal: 70% stated, 70% observed.

• A single point below the diagonal: that bucket's stated numbers were too high for what happened. (Both of the broken buckets above.)
• A single point above the diagonal: that bucket's stated numbers were too low; things happened more often than the forecaster was willing to say.
• A curve flatter than the diagonal, crossing it near the middle: the pattern usually described as overconfidence. It sits above the line at the low end and below it at the high end: a 10% that resolved yes 25% of the time and a 90% that resolved yes 75% of the time are the same tendency measured from opposite ends, with the probabilities pushed too far toward 0 and 1. The middle of the range typically tracks better.

Read the points against the diagonal; there's no separate benchmark. The example log above has buckets from 50% up, so it shows the high-end half of that pattern only. Buckets on both sides of the middle are needed to see the whole shape.

Calibration is not accuracy, and not a Brier score

Three different questions, routinely confused:

The three can point in different directions on the same set of forecasts. A forecaster who states 55% on everything, and is right that 55% of them occur, is perfectly calibrated with a middling Brier score: the stated probabilities match the outcomes, and they carry little information, because none of them committed to much. The reverse also happens: a reasonable Brier score can sit on top of one badly-off bucket that the average absorbs. Calibration is generally used as the diagnostic and Brier as the summary figure.

Sample size: why 20 forecasts tell you almost nothing

Take the 80–89% bucket above: 12 forecasts. Had two of them gone the other way, the hit rate would read 83% rather than 67%, and the bucket would look calibrated. Two outcomes move it sixteen points.

That's the arithmetic of small samples rather than a problem with the method, and it's why a calibration table built on a handful of forecasts can present a precise-looking percentage that would move substantially on one or two different outcomes. As a rough guide, a bucket with fewer than about 20 resolved forecasts is usually treated as suggestive rather than conclusive, and covering four or five buckets at that depth typically takes a few hundred forecasts in total.

A calibration result is generally only interpretable alongside its sample size, which is usually the first thing worth looking for when one is published.

Calibrating yourself

The procedure that makes a log scoreable:

• Write the number down, dated, before resolution. A probability recalled afterwards has usually drifted toward what happened, so it can't be scored accurately.
• Use round buckets. 60%, 70%, 80%. Finer gradations than you can actually distinguish add logging effort without adding signal.
• Score in batches rather than one at a time. A single outcome is consistent with almost any stated probability; a bucket of twenty or so starts to constrain what the stated number could reasonably have been.
• Start from the base rate, how often the thing tends to happen in general, and adjust from there. Anchoring on a specific narrative rather than a frequency is one commonly described route to a miscalibrated number.
• Then compute a Brier score on the same log, so you have the summary figure alongside the diagnostic.

Reading someone else's calibration claim

The same three questions apply to any published calibration figure, whoever produced it: which set of questions was scored, at what horizon, and over how many resolved forecasts. Whether prediction market prices are well calibrated is an open empirical question rather than a settled one. Market efficiency covers the underlying hypothesis and the conditions under which it holds up better or worse.