Expected calibration error: the confidence definition
Guo and colleagues’ 2017 paper describes an empirical ECE that groups predicted-label confidence and weights each bin’s absolute accuracy-confidence gap by its fraction of samples. In symbols, ECE = Σ (nᵦ / N) × |accuracyᵦ − confidenceᵦ|. The paper’s experiments do not measure prediction-market or Alphascope performance.
This binary lesson predicts Yes at probability at least 50%, otherwise No. Confidence is max(p, 1 − p). A 20% Yes forecast therefore predicts No with 80% confidence; its correctness depends on whether No occurred. The tool’s lower-inclusive bin convention is declared above and differs from the paper’s upper-inclusive intervals. Specify your convention when comparing results.
A reliability diagram for Yes probabilities
In the alternate interpretation, keep p as the probability of Yes and compare each bin’s mean p with its observed Yes frequency. This view can reveal which side of an event forecast is off. It does not replace p with confidence in the predicted label.
For an original two-record example, enter 20,0 and 80,0 with ten bins. Confidence mode pools both 80%-confident labels: one correct and one incorrect. Its gap is 30 percentage points. Yes-probability mode separates 20% and 80% forecasts, with observed Yes frequency zero in each. Its weighted gap is 50 percentage points. The difference is a change in the question, not a calculation error.
Why the number of bins matters
Consider 40,1 and 60,0 in Yes-probability mode. With one bin, the mean probability and observed Yes frequency are both 50%, so the summary gap is zero. With two bins, each contains one record and the weighted gap is 60 percentage points. A zero coarse-bin result can hide large opposing errors.
Choose and report the bin rule before using a score to compare models. Do not search for whichever bin count makes your model look best. Empty bins provide no evidence; a one-record bin is also a weak basis for claims about a population. The chart plots occupied-bin means rather than fabricating points for empty ranges.
Interpret percentage points correctly
A mean confidence of 80% and observed accuracy of 75% differ by five percentage points. If that bin contains two thirds of the records, it contributes about 3.333 percentage points to the weighted summary. This is an absolute probability gap, not a 3.333% trading return or a guarantee about the next forecast.
The default fictional six-row example has top-label ECE of about 3.333 percentage points and a Yes-probability gap of about 16.667 percentage points with ten bins. Inspect its table rather than treating either total as a complete assessment of forecast quality.
What this calculator does not establish
It does not fit a new forecasting model, validate resolutions, estimate confidence intervals or prove that repeated observations are independent. Preserve timestamps, market IDs, outcome definitions and the forecast horizon. Keep losing forecasts, report exclusions and compare the same resolved sample.
To evaluate probability performance from a different angle, use the log-loss calculator and Brier score calculator. Neither score alone isolates calibration. Forecast quality also differs from trading profitability after prices, fees and execution.
Calibration diagnosis versus calibration fitting
For a worked fitting workflow, read our Platt scaling and isotonic calibration guide and run its synthetic Python example.
Platt scaling and isotonic regression are ways to fit a probability mapping, not synonyms for ECE. The scikit-learn calibration guide distinguishes evaluating reliability from fitting calibrated classifiers. Keep fitting and evaluation data separate when assessing an improvement; this tool applies no fitted transformation.
Sources reviewed October 13, 2026. These calculations are descriptions of entered binary records, not claims that a particular platform or AI model is calibrated.