We trained on 36,000 matches. The model fell apart in May.

We trained on 36,000 matches. We expected the model to struggle with upsets. What we didn't expect was that it would fall apart specifically in May.

When we cut our evaluation data by matchday, a pattern appeared that forced us to rethink one of the model's core assumptions: accuracy at matchday 37–38 of a 38-game season dropped to around 26%. Mid-season — matchdays 10–25 — it sat at roughly 41%. The model wasn't worse at football. It was worse at end-of-season football. Those are different things.

What changes in May

Three things happen simultaneously in the final weeks of a European league season, and our initial model treated none of them as signals.

Squad rotation.A club that's already secured the title, or is mid-table with nothing to play for, rests first-team players. The expected-goals model built from the last ten matches is still reading Erling Haaland's numbers, but Haaland isn't starting. The form signal is stale.

Motivation asymmetry.Two teams that looked balanced on paper in February might arrive at matchday 37 in wildly different mental states: one fighting relegation with everything, the other with a week's worth of beach flights already booked. Our feature set had no way to capture that asymmetry. Form and H2H don't tell you how much a result matters.

Tactical divergence.Teams chasing safety tend to defend deep and play on the counter. Teams that have nothing to lose start youngsters. The distributional shift from “normal season football” to “fixture 37 of 38” is large enough to blunt features calibrated on the bulk of the season.

What we added

The solution we shipped — which we call season_completion_pct— is a single number: matchday divided by the league's maximum matchdays, clamped between 0 and 1. EPL and La Liga top out at 38. Bundesliga and Ligue 1 at 34. So matchday 38 in the Premier League maps to a completion score of 1.0; matchday 10 maps to 0.26.

That number lets XGBoost learn that its other features — form, Poisson goal estimates, H2H — carry different predictive weight depending on where you are in the season. A 60% home-win probability in October and a 60% home-win probability in May mean different things when one of the teams involved is already safe.

We also added what we're calling motivation proxies: cumulative points in the current season, points-per-game, estimated distance to a top-four finish, estimated distance from the relegation zone, and remaining points possible. These are computed entirely from results in the fixtures table — no Sportmonks standings backfill needed.

How much did it help

Enough to promote a new model. Our ECE (Expected Calibration Error — how far off our confidence estimates are on average) improved from 0.041 to 0.039. Brier score held steady at 0.210. Not a revolution. But calibration improvement at this scale, across 36,000 matches, is meaningful: it means our 60% predictions are actually closer to winning 60% of the time.

The accuracy gap at matchday 37–38 narrowed by about 5 percentage points in held-out evaluation. That's still not “good at end-of-season football” — we don't have confirmed lineup data, and we never will from our data source. But it's honest about the uncertainty: those matches now come out of the model with appropriately wider probability distributions, which the uncertainty index on each match card reflects.

What we can't fix without better data

Some of the end-of-season prediction problem is irreducible without real-time lineup information. If we knew with certainty that a manager was resting four starters, the model's confidence interval would collapse appropriately. We don't have that.

What we can do is be honest about it. The Hard to Call card — which fires when entropy is high, draw probability is elevated, or maximum probability is below 42% — tends to appear disproportionately in May. That's not a bug. The model is accurately describing its own uncertainty rather than manufacturing false confidence.

The broader lesson

The most useful thing we extracted from the matchday analysis wasn't the feature. It was the reminder that “football data” is not a single distribution. A Premier League fixture in October and a Premier League fixture in May are played under different conditions, with different motivations, by partially different squads. A model that pretends otherwise is training on a fiction.

We'll keep adding signals that capture the conditions under which a match is actually played — not just the historical record of the teams playing it. You can track how the model improves (or doesn't) on the Track Record page. The ECE and Brier numbers update as live results come in.