Model Evaluation Report: Sample
What an honest model evaluation looks like before a go-live decision, baseline first, performance by segment, the failure analysis, the limitations section, and a recommendation that names its conditions.
Markdown. No sign-up, no email.
This is an illustrative example. The model, figures and findings are composed to show the shape of an honest evaluation, not drawn from a named engagement.
Model: Delivery delay prediction v1.4 Prepared by: _______ Date: _______ Decision requested: approve for production use in the operations queue
1. Summary#
The model predicts whether an order will miss its promised delivery date, 24 hours ahead.
Recommendation: approve, for the ranked-queue use case only, with monitoring in place before go-live. It is not suitable for automated customer notification, for the reason in §5.
| Recall (missed deliveries found) | 0.71 |
| Precision | 0.64 |
| Trivial baseline recall | 0.09 (base rate) |
| Business threshold agreed beforehand | recall ≥ 0.60 |
2. Baseline first#
| Approach | Recall | Precision |
|---|---|---|
| Always predict "on time" | 0.00 | |
| Existing rule (no scan in 18h) | 0.44 | 0.51 |
| This model | 0.71 | 0.64 |
The existing rule is the honest comparison, not the do-nothing baseline. The model finds roughly 27 percentage points more of the delays that occur, at better precision.
3. Method#
- Split: by time. Trained to 31 March, validated April, tested May onwards
- Test set used once, at the end
- Features verified answerable as of the prediction timestamp
- Leakage review completed; two features removed (both populated after delivery)
- Results reproducible from a clean run; seed and library versions pinned
4. Performance by segment#
Aggregate figures hide the thing that matters.
| Segment | Volume share | Recall | Note |
|---|---|---|---|
| Standard domestic | 68% | 0.78 | strongest |
| Next-day domestic | 14% | 0.69 | |
| International | 11% | 0.41 | weak |
| Bulk / palletised | 5% | 0.33 | very weak |
| New customers (<30 days) | 2% | 0.38 | little history |
The model is substantially worse on international and bulk shipments, under half the delays found, and those are the consignments where a delay costs most.
This is the finding that shaped the recommendation.
5. Failure analysis#
Where it fails: consignments with sparse tracking history. The model leans on scan-event frequency, and carriers on international and bulk routes scan less often. Absence of a scan means "late" domestically and "normal" internationally, and the model has not learned that distinction well.
How it fails: silently and confidently. A missed prediction is indistinguishable from a correct "on time" call.
Why this rules out customer notification: telling a customer their delivery is late when it is not, at 0.64 precision, means roughly one in three notifications is wrong. For the internal queue that is acceptable: an operator checks. For a customer message it is not.
6. Limitations#
- Trained on 14 months. No full peak season in the training window; December behaviour is unproven
- Two carriers added in April are under-represented
- Assumes current scan-event frequency; a carrier changing its scanning practice degrades this silently
- Not evaluated for fairness across customer segments, not obviously relevant here, not checked
7. Cost and latency#
| Inference cost | £0.0004 per prediction |
| At current volume | ~£118/month |
| p95 latency | 240 ms |
| Retraining | ~4 hours, monthly |
8. Conditions of approval#
- Used to rank the operations queue, never to notify customers automatically
- International and bulk consignments flagged as low-confidence in the interface
- Outcome capture in place before go-live, so production accuracy is measurable
- Alert if weekly recall falls below 0.60
- Reviewed after the first December, before which peak behaviour is unknown
9. What would change this recommendation#
Better recall on international and bulk: most likely from carrier scan data rather than from model changes. If that data becomes available, re-evaluate; the current ceiling is a data limitation, not a modelling one.
Why this report is shaped this way. It leads with the baseline, reports by segment rather than in aggregate, states where and how the model fails, and makes a recommendation with named conditions. A report that gives a single accuracy figure and a recommendation invites approval without any of the information that should qualify it.
See machine learning and the model review checklist.