Calibration
Updated November 10, 2026
Calibration is how often events happen when you (or a market) said they would: 70% forecasts should resolve yes about 70% of the time over many cases. Well-calibrated forecasts match outcomes; miscalibration shows overconfidence or underconfidence.
You log 20 forecasts at ~70%. 14 resolve yes (a 70% hit rate), so you were well calibrated on that bucket. If only 9 hit (45%), you were overconfident at 70%. A market's prices can be scored the same way.
- Resolve yes: well calibrated
- 14 of 20
- 70% hit rate
- Resolve yes: overconfident at 70%
- 9 of 20
- 45% hit rate
How to check calibration
Write each forecast down, dated, before the outcome is known. Once enough have resolved, group them into buckets by stated probability (the 50s, 60s, 70s and 80s) and compare each bucket’s hit rate with its label. A bucket at 70–79% should resolve yes roughly 70–79% of the time.
Plotted as stated probability against observed hit rate, a perfectly calibrated set sits on the 45° diagonal. A point below the diagonal means the stated numbers in that bucket were too high: overconfidence at that level. A point above it means they were too low. The chart points at a range, which is more useful than a general verdict on the forecaster.
The same check works on a market. Take each contract’s implied probability at a fixed time before resolution, such as a week out, and bucket those prices instead of your own numbers. Contracts priced around 70¢ should resolve yes about 70% of the time across many markets.
Why 20 forecasts is a small sample
In the worked example, each forecast that goes the other way moves the hit rate by 5 points, because one forecast is 1/20 of the bucket. Two different outcomes would turn the calibrated 14 of 20 (70%) into 12 of 20 (60%). That is why a bucket with fewer than about 20 resolved forecasts is usually read as suggestive, and why checking four or five buckets at that depth takes a few hundred forecasts in total.
Calibration vs Brier score
A Brier score answers a different question. Calibration is a pattern across buckets; the Brier score is one number for overall error. A forecaster who says 55% on everything, and sees 55% of those events happen, is perfectly calibrated with a middling Brier score, because none of the forecasts committed to much. Calibration in depth works through a 60-forecast log.
FAQ
What does it mean for a forecast to be well calibrated?
That its stated probabilities match observed frequencies: events called 70% happen about 70% of the time, events called 30% about 30% of the time, across many forecasts.
How do you measure calibration?
Bucket resolved forecasts by stated probability and compare each bucket’s hit rate with its label. Plotting the buckets against the 45° diagonal shows where the forecasts ran too high or too low.
Are prediction markets well calibrated?
It depends on the set of questions, the horizon and the sample size, so it has to be measured rather than assumed. The method is the same as for a person: bucket the prices at a fixed time and compare them with outcomes.