Machine Learning — Frequently Asked Questions
Practical answers on building ML systems — how much data you need, why your validation score does not survive production, class imbalance, drift, MLOps, and when a language model is the wrong tool.
Before you build#
How much data do we need?#
Less than people fear for simple problems, more than they hope for hard ones. A well-defined binary classification with clear signal can work with a few thousand labelled examples; anything subtle, multi-class or rare-event needs far more.
The better question is how much labelled data you have, and whether the labels are consistent. Two annotators disagreeing on a third of cases caps your model's achievable accuracy no matter how much data you add.
Do we still need classical ML now that language models exist?#
Yes, for a large class of problems. Tabular data — churn, fraud, pricing, demand, risk scoring — is still handled better, faster and far more cheaply by gradient-boosted trees than by anything generative. They also explain themselves better and cost a fraction to run.
Reach for a language model when the input is language or the task needs world knowledge. Reach for classical ML when the input is rows and columns.
What baseline should we beat?#
The dumbest thing that could work: predict the majority class, predict last week's value, or keep the existing business rule. Measure it properly.
Teams routinely skip this and celebrate 85% accuracy on a problem where always answering "no" scores 84%. The baseline is what tells you whether the model is worth its operational cost.
Why models fail#
Our model scored 95% in testing and fell apart in production. Why?#
Most likely leakage — a feature containing information that would not be available at prediction time. Something derived from the target, a field populated after the event, or a random split on time-ordered data letting the model see the future.
The tell is a score that seems too good. Treat that as a symptom, not a success.
What is training-serving skew?#
Features computed one way for training and a slightly different way in production. It happens whenever the two paths are separate implementations, and it produces a model that is subtly wrong in ways nothing catches, because both halves individually look correct.
Share the feature computation code between training and serving. Where you cannot, test that both produce identical output on the same input.
Our accuracy is high but the model is useless. What went wrong?#
Almost certainly class imbalance. On a dataset where 1% of cases are fraud, a model that predicts "not fraud" every time scores 99%.
Use metrics that reflect the consequence — precision and recall on the class you care about — and cost false positives and false negatives separately, because they are rarely equally expensive.
The model worked for six months and then degraded. Why?#
Drift. The input distribution moved, or the relationship between inputs and outcome changed — new products, a pricing change, a competitor, a season, a pandemic.
This is why models need monitoring on inputs, predictions and actual outcomes. A model with no outcome capture cannot be measured in production at all; you are flying on the evaluation score from launch day forever.
Building and running#
What is the minimum viable MLOps?#
Four things: reproducible training from a clean checkout, versioned models with the version recorded on every prediction, monitoring that captures actual outcomes, and a tested rollback to the previous model.
Everything else — feature stores, registries, orchestration platforms — is worth adding when you feel its absence, not before. Most teams need the four above and buy the rest first.
How often should we retrain?#
When something triggers it: measured drift, a performance drop, or a known change in the business. Retraining on a fixed schedule with no trigger burns compute and occasionally makes things worse without anyone noticing.
Always compare the candidate against the incumbent on a held-out set before replacing it.
Should we use deep learning?#
For images, audio, video and language, yes — that is where it wins decisively. For tabular business data, usually not: tree ensembles typically match or beat it, train in minutes rather than hours, and need far less tuning.
Choose by data type, not by novelty.
How do we explain a prediction?#
Decide whether you need global explanation (what drives the model overall) or local (why this prediction). Feature importance covers the first; per-prediction attribution methods cover the second.
Be honest about the limits: these are approximations of the model's behaviour, not the reasoning itself. If a regulator or a customer needs a defensible explanation, that requirement should shape your model choice from the start — a simpler, inherently interpretable model is often the right trade.
How do we know when to stop improving it?#
When the next point of accuracy is worth less than the effort to get it. Fix that threshold before starting, because model improvement is unbounded and comfortable work, and it is easy to spend a quarter moving a number that changes no decision.
Organisation#
Who should own a production model?#
The team that owns the decision it feeds, with data science support. A model owned solely by a central team becomes an unmaintained dependency the moment its author moves on.
What is the most common organisational mistake?#
Treating a model as a project rather than a system. It ships, the team disbands, nobody monitors it, and eighteen months later it is quietly making worse decisions than the rule it replaced — with more confidence and less scrutiny.