Calibration, explained for the data-curious fan

Here's a sentence that sounds obvious until you sit with it: if a probability model says “60% home win”, then across all the fixtures where it said that, the home team should actually win about 60% of the time.

That property is called calibration. It's the difference between a model that's telling you something useful and a model that's just making confident noises. Most published football prediction models — including some of the well-known commercial ones — fail this test. Here's why, and what to do about it.

The thing you really want from a probability

Take 1,000 fixtures where the model stated the home team had a 60% chance. Three numbers matter:

• How often the home team actually won across those 1,000 fixtures
• What the model claimed (60%)
• The gap between the two

A perfectly calibrated model has zero gap. A model that claims 60% and the home team wins 50% of the time is over-confident: it's promising more certainty than reality delivers. A model that claims 60% and the home team wins 70% of the time is under-confident: it sees a clearer pattern than its numbers suggest.

Both errors are common. Over-confidence is the killer, though, because it gives users a false sense of edge.

Why most ML models aren't calibrated out of the box

XGBoost, LightGBM, neural nets, ensemble models — none of them produce calibrated probabilities by default. They produce scoresthat sort the predictions in roughly the right order (the model knows Arsenal is more likely to beat Brighton than vice versa) but the scores themselves don't map to real-world frequencies.

The fix is a second model on top. A small one — typically isotonic regression or Platt scaling — fits the model's raw scores against observed outcomes on a held-out set, then becomes a translator: input score 0.83, output calibrated probability 0.62. The combined system is calibrated.

It's a small piece of machinery that takes about an afternoon to implement properly. The number of football prediction sites that skip this step entirely is depressing.

How to test calibration yourself

The visual test is the reliability diagram. Bin the model's predictions by their stated probability (0-10%, 10-20%, ... , 90-100%). For each bin, compute the actual frequency of the outcome in question. Plot predicted vs observed; if calibration is good, the points fall on the diagonal.

The numeric test is Expected Calibration Error (ECE). It's the average gap between predicted and observed across all bins, weighted by how many predictions are in each bin. Lower is better. A well-calibrated football 1×2 model lands in the 0.03–0.06 range; uncalibrated models routinely sit at 0.10+.

What MatchMind does

We run an isotonic-regression calibration layer on every published prediction. The current champion model's validation ECE is on the Track Record page — published openly so you can sanity-check.

We also keep a rolling live-evaluation of every pre-match prediction (`monitoring.prediction_evaluations`) so we can re-render the reliability diagram from real outcomes, not just held-out training data. Once we have 50+ live evaluations, that curve goes on the public page.

What calibration doesn't get you

Honest caveats:

• Calibrated ≠ accurate. A model that always predicts 33% is perfectly calibrated to a random outcome, and tells you nothing. You need calibration AND skill (low Brier, low log-loss).
• Calibration shifts.A new manager, a new tactical era, a rule change — any of these can shift the underlying distribution. Last season's calibrated model is this season's drift problem. We re-train periodically and monitor for drift in trend signals.
• Calibration is a population property, not a per-fixture property.A 60% probability doesn't mean “the home team will win 60% of this match.” That sentence doesn't mean anything. It means: across all matches we'd call 60%, the home team wins 60% of them. A single match wins or loses; the probability is about the distribution.

The headline

Calibration is the unglamorous, easy-to-verify property that separates probability models you can trust from probability models you can't. It's also the property most football sites quietly skip — because showing your reliability diagram means committing to be measured.

Read ours → matchmind.dev/track-record