Data Leakage — A Worked Example
A churn model that scored 94% and was worthless, traced feature by feature — what leaked, how it was found, what the honest score turned out to be, and what the project did next.
A subscription business builds a churn model. The brief: predict which customers will cancel in the next 30 days, so the retention team can call them.
The numbers below are illustrative, but the shape of this failure is one of the most common in applied machine learning.
The result that should have raised suspicion#
First model, gradient-boosted trees, 40 features from the warehouse:
| Metric | Value |
|---|---|
| Accuracy | 94% |
| Precision (churn class) | 0.89 |
| Recall (churn class) | 0.86 |
| Baseline — always predict "no churn" | 91% accuracy |
The team was pleased. Notice the last row: the trivial baseline scores 91%, because only 9% of customers churn. The model's headline accuracy is three points above doing nothing at all.
That is the first signal, and it is a signal about the metric. Precision and recall on the churn class are what matter, and at 0.89 / 0.86 they looked genuinely strong.
The second signal#
Feature importance, top five:
| Rank | Feature | Importance |
|---|---|---|
| 1 | days_since_last_login | 0.31 |
| 2 | support_tickets_30d | 0.18 |
| 3 | cancellation_reason_code | 0.16 |
| 4 | plan_downgraded | 0.11 |
| 5 | monthly_spend_change | 0.09 |
Number three should stop everything. A cancellation reason code exists because somebody cancelled. The model had learned to predict churn from a field populated by churn.
What leaked, and how#
Three separate leaks, discovered in order:
1. The obvious one. cancellation_reason_code is null for active customers and populated at cancellation. The model learned "field is not null → churn". Removing it dropped recall to 0.71.
2. The subtle one. support_tickets_30d was computed as a rolling window from the warehouse at query time, not as of the prediction date. For a customer who churned on 14 March, the window included tickets raised while they were cancelling. Recomputing it as of the prediction date dropped recall to 0.58.
3. The structural one. The train/test split was random. Customers appear in the data monthly, so the same customer's January row was in training and their February row in test. The model was not generalising to new customers; it was recognising ones it had seen. Splitting by time — train on data up to December, test on January onwards — dropped recall to 0.44.
The honest number#
| Version | Recall (churn) | Precision |
|---|---|---|
| Original | 0.86 | 0.89 |
| Without the cancellation field | 0.71 | 0.77 |
| With point-in-time features | 0.58 | 0.69 |
| With a time-based split | 0.44 | 0.61 |
The real model finds slightly under half of churners, and about six in ten of the customers it flags do churn.
Was that still useful?#
Yes — and this is the part that matters. The retention team had capacity to call roughly 200 customers a month. They did not need to catch every churner; they needed a better-than-random list of 200.
At 0.61 precision, calling the model's top 200 reaches around 122 genuine churners, against roughly 18 from calling 200 customers at random. That is a real improvement, and it was deliverable — whereas the 94% model would have been deployed, would have failed to find churners it had never been able to find, and would have discredited the whole programme.
What they changed#
- Time-based splits by default, on every model
- A rule that any feature must be answerable as of the prediction timestamp
- Feature importance reviewed before results are presented, not after
- The trivial baseline reported alongside every model score
That last one is the cheapest control here. It reframes "94% accurate" as "three points above doing nothing", which is the conversation worth having.
See machine learning and the review checklist, whose leakage section exists because of failures shaped exactly like this.