# Model Evaluation Report — Sample > This is an illustrative example. The model, figures and findings are composed to show the shape > of an honest evaluation, not drawn from a named engagement. **Model:** Delivery delay prediction v1.4 **Prepared by:** _______ **Date:** _______ **Decision requested:** approve for production use in the operations queue ## 1. Summary The model predicts whether an order will miss its promised delivery date, 24 hours ahead. **Recommendation: approve, for the ranked-queue use case only, with monitoring in place before go-live.** It is not suitable for automated customer notification, for the reason in §5. | | | |---|---| | Recall (missed deliveries found) | 0.71 | | Precision | 0.64 | | Trivial baseline recall | 0.09 (base rate) | | Business threshold agreed beforehand | recall ≥ 0.60 | ## 2. Baseline first | Approach | Recall | Precision | |---|---|---| | Always predict "on time" | 0.00 | — | | Existing rule (no scan in 18h) | 0.44 | 0.51 | | **This model** | **0.71** | **0.64** | The existing rule is the honest comparison, not the do-nothing baseline. The model finds roughly 27 percentage points more of the delays that occur, at better precision. ## 3. Method - **Split:** by time. Trained to 31 March, validated April, tested May onwards - **Test set used once**, at the end - **Features** verified answerable as of the prediction timestamp - Leakage review completed; two features removed (both populated after delivery) - Results reproducible from a clean run; seed and library versions pinned ## 4. Performance by segment Aggregate figures hide the thing that matters. | Segment | Volume share | Recall | Note | |---|---|---|---| | Standard domestic | 68% | 0.78 | strongest | | Next-day domestic | 14% | 0.69 | | | International | 11% | **0.41** | weak | | Bulk / palletised | 5% | **0.33** | very weak | | New customers (<30 days) | 2% | 0.38 | little history | The model is substantially worse on international and bulk shipments — under half the delays found — and those are the consignments where a delay costs most. This is the finding that shaped the recommendation. ## 5. Failure analysis **Where it fails:** consignments with sparse tracking history. The model leans on scan-event frequency, and carriers on international and bulk routes scan less often. Absence of a scan means "late" domestically and "normal" internationally, and the model has not learned that distinction well. **How it fails:** silently and confidently. A missed prediction is indistinguishable from a correct "on time" call. **Why this rules out customer notification:** telling a customer their delivery is late when it is not, at 0.64 precision, means roughly one in three notifications is wrong. For the internal queue that is acceptable — an operator checks. For a customer message it is not. ## 6. Limitations - Trained on 14 months. No full peak season in the training window; December behaviour is unproven - Two carriers added in April are under-represented - Assumes current scan-event frequency; a carrier changing its scanning practice degrades this silently - Not evaluated for fairness across customer segments — not obviously relevant here, not checked ## 7. Cost and latency | | | |---|---| | Inference cost | £0.0004 per prediction | | At current volume | ~£118/month | | p95 latency | 240 ms | | Retraining | ~4 hours, monthly | ## 8. Conditions of approval 1. Used to **rank** the operations queue, never to notify customers automatically 2. International and bulk consignments flagged as low-confidence in the interface 3. Outcome capture in place **before** go-live, so production accuracy is measurable 4. Alert if weekly recall falls below 0.60 5. Reviewed after the first December, before which peak behaviour is unknown ## 9. What would change this recommendation Better recall on international and bulk — most likely from carrier scan data rather than from model changes. If that data becomes available, re-evaluate; the current ceiling is a data limitation, not a modelling one. --- **Why this report is shaped this way.** It leads with the baseline, reports by segment rather than in aggregate, states where and how the model fails, and makes a recommendation with named conditions. A report that gives a single accuracy figure and a recommendation invites approval without any of the information that should qualify it. See [machine learning](/knowledge/machine-learning/) and the [model review checklist](/knowledge/machine-learning/checklist/).