Machine Learning Model Review Checklist
Run before a model goes to production, leakage, the validation split, class imbalance, the baseline it must beat, drift monitoring, and who is accountable for its decisions.
Markdown. No sign-up, no email.
Model: _______________ Owner: _______________ Date: _______
1. The problem#
- [ ] The prediction target is defined precisely, including edge cases
- [ ] A trivial baseline exists and has been measured. Always predict the majority class, predict yesterday's value, use the existing rule
- [ ] The model beats that baseline by a margin worth the operational cost
- [ ] The decision made from the prediction is documented
Baseline score: ____ Model score: ____ Difference worth the complexity? yes / no
A model that beats a trivial baseline by two points is usually not worth the pipeline, monitoring and on-call that come with it. Measure the baseline first, not last.
2. Data leakage: the defect that produces excellent results and useless models#
- [ ] No feature contains information unavailable at prediction time
- [ ] No feature derived from the target
- [ ] Time-ordered data split by time, never randomly
- [ ] Preprocessing fitted on training data only, then applied to validation
- [ ] Duplicates removed before splitting, not after
- [ ] Groups (customer, device, session) kept entirely within one split
Suspiciously high score? Assume leakage before assuming success. It is the single most common cause of a model that performs brilliantly in evaluation and fails immediately in production.
3. Validation#
- [ ] Held-out test set, used once, at the end
- [ ] Validation strategy matches how the model will be used
- [ ] Cross-validation where data is limited
- [ ] Performance reported with a range, not a single number
- [ ] Results reproducible from a clean run, seed and versions pinned
4. Metrics#
- [ ] The metric matches the business consequence
- [ ] Class imbalance accounted for. Accuracy on a 99:1 split is meaningless
- [ ] False positives and false negatives costed separately
- [ ] Threshold chosen deliberately, not left at 0.5
- [ ] Performance checked per segment, not only in aggregate
| Segment | Score | Volume |
|---|---|---|
An aggregate score can hide a model that fails badly for a specific group. Check before someone else does.
5. Fairness and explanation#
- [ ] Checked for disparate performance across groups where relevant
- [ ] Proxies for protected characteristics considered (postcode, education, gaps)
- [ ] Can explain an individual prediction if challenged
- [ ] Decisions with legal or significant effect on people have human review
6. Production#
- [ ] Training and serving use the same feature computation. A separate reimplementation is the classic source of silent skew
- [ ] Input validation at the boundary; behaviour defined for out-of-range values
- [ ] Latency and cost per prediction measured
- [ ] Model version recorded with every prediction
- [ ] Predictions logged with inputs for later analysis
- [ ] Fallback when the model is unavailable
7. Monitoring#
- [ ] Input distribution monitored for drift
- [ ] Prediction distribution monitored
- [ ] Actual outcomes captured, so accuracy can be measured in production
- [ ] Alert thresholds set, routed to a named person
- [ ] Scheduled review, because the world changes even when the code does not
How will we know this model has degraded? _______________
If the answer is "someone will notice", it will degrade unnoticed for months.
8. Retraining#
- [ ] Trigger defined, schedule, drift, or performance drop
- [ ] Retraining is reproducible
- [ ] New model compared to current before replacing it
- [ ] Rollback to the previous model tested
Sign-off#
| Name | Date | |
|---|---|---|
| Built by | ||
| Reviewed by | ||
| Approved for production |