Machine Learning: When It Beats Simple Rules, and When It Doesn't

Machine learning finds patterns in historical data and uses them to make predictions about new cases. Which customers are likely to leave. Which transactions look fraudulent. What demand will be next month.

The important question is not whether it works. It is whether it beats the simple thing you would otherwise do — and that comparison is skipped far more often than it is made.

Start with the baseline#

Before any model, write down what a simple rule achieves.

Predicting customer churn? The baseline might be "customers who have not logged in for 60 days". Forecasting demand? "Same month last year, adjusted for growth."

Then measure the model against that. A model that is 3% better than a rule anyone can explain, while requiring a pipeline, monitoring and a specialist to maintain, is usually a bad trade. A model that is 30% better is transformative.

Most disappointing ML projects never established the baseline, so nobody could tell which of those two situations they were in.

What ML actually needs#

Enough historical examples, with outcomes. You need cases where you know what happened. Not "customer data" — labelled data. Thousands of examples for simple problems, more for hard ones.

The outcome must be recorded. To predict which orders will be returned, you need historical returns linked to orders. Organisations frequently discover their most valuable label was never captured.

The future must resemble the past. ML learns historical patterns. If your market, pricing or customer base changed fundamentally, the model learns a world that no longer exists.

Someone must be able to act on the prediction. Knowing a customer will probably churn is worth nothing unless something happens as a result.

If any of these is missing, no algorithm rescues the project.

The problems it suits#

Notice these are narrow, repeated decisions with measurable outcomes. That is the sweet spot: the same question asked thousands of times.

How models fail in production#

Different from how software fails, and less obvious.

Drift. The world moves; the model does not. Accuracy decays gradually, often without anyone noticing, because nothing errors. A model that was 85% accurate at launch might be 70% a year later and still be producing confident predictions.

Training-serving skew. The data the model sees in production differs subtly from training data — a field formatted differently, a default value changed. Performance drops for reasons that look like nothing.

Leakage. The model learned from information that would not be available at prediction time. Symptom: suspiciously excellent test results, disappointing production results. If a model looks too good, suspect leakage before celebrating.

Feedback loops. The model's predictions change behaviour, which changes the data, which changes the model. A fraud model that blocks certain transactions stops seeing whether they were fraudulent.

Optimising the wrong thing. A 99% accurate model on a problem where 99% of cases are negative has learned to say "no" every time. Accuracy is the most misleading metric in common use — for imbalanced problems, look at precision and recall separately.

Measuring what matters#

Pick the metric that matches the business cost of each error type.

If missing a fraudulent transaction costs far more than reviewing a legitimate one, you want high recall and will accept false positives. If wrongly flagging a customer damages the relationship, you want high precision. These trade against each other; the model does not know which you care about unless you say.

Then translate to money. "Precision 0.82" means nothing to a sponsor. "Catches 4 in 5 fraud cases, flags 200 legitimate transactions a month for review, net saving X" is a decision.

Fairness, plainly#

If a model is trained on historical decisions, it learns historical patterns — including discriminatory ones. Removing a protected attribute does not fix this, because other fields often predict it.

Practical minimum: test performance separately across the groups that matter, before deployment. If accuracy differs materially between groups, you have a problem to address rather than a curiosity to note. For consequential decisions — credit, employment, housing, insurance — this is increasingly a legal obligation, not an ethical preference.

A realistic path#

Frame the decision first. What gets decided, by whom, how often, and what does an error cost?

Check the data honestly. Do you have labelled outcomes? How far back? How clean?

Build the baseline. Simple rule, measured properly. Sometimes you stop here, and that is a success — you solved the problem cheaply.

Build a simple model. Logistic regression or gradient boosting on tabular data beats deep learning for most business problems and is far easier to explain and operate.

Test on data from a later period than training, never a random split of the same period. Random splits flatter models by leaking future information.

Deploy alongside the current process. Compare for a few weeks before anything depends on it.

Monitor from day one. Input distributions, prediction distributions, and accuracy where outcomes eventually arrive. Without this you will not notice drift.

FAQ#

What is the difference between machine learning and AI?#

Machine learning is one approach within AI: learning patterns from data. The language models behind current AI assistants are machine learning too, just a particular kind trained on text at very large scale. For business purposes, "ML" usually means models predicting from your own structured data.

How much data do we need?#

For a straightforward classification problem with clean tabular data, thousands of labelled examples can be enough. Complex problems need much more. More useful than the count: are the examples representative, are the labels correct, and do they cover the cases you care about?

Can we use ML without a data scientist?#

Automated ML tools have made simple models accessible. What still needs expertise is framing the problem, spotting leakage, choosing the right metric, and knowing when the model is quietly wrong. Those are the parts that cause expensive mistakes.

How often should a model be retrained?#

Depends on how fast your world changes — monthly is common, but the honest answer is "when monitoring says performance has dropped". Retraining on a fixed schedule without measuring is a ritual, not a control.

Why did our model perform worse in production?#

Usually leakage in training, training-serving skew, or a test set that shared time periods with training. All three make a model look better than it is before it meets reality.

Should we use deep learning?#

For images, audio, video and free text, yes. For rows and columns — which is most business data — gradient-boosted trees usually match or beat it, train in minutes rather than hours, and are far easier to explain.

Can we explain a model's decisions?#

Simple models, yes, directly. Complex models can be explained approximately with established techniques. If you need to justify individual decisions to a regulator or a customer, prefer a simpler model from the start — retrofitting explainability is harder than choosing for it.

What else is coming for Machine Learning