BvLogic
Diagram · Machine Learning

Training-Serving Skew — Diagram

The two paths a feature travels, drawn side by side — why they drift apart, the four places skew enters, and the single change that removes most of it.

Download the diagram Read the full guide

SVG. No sign-up, no email.

A model is trained on features computed one way and serves predictions on features computed another. When those two computations differ even slightly, the model is subtly wrong in production while both halves individually look correct.

Nothing fails. No error is raised. Accuracy is simply lower than evaluation promised, and the investigation goes to the model — which is the wrong place.

One feature, two paths Training · batch: Warehouse table (complete, corrected,,late-arriving rows settled) → Aggregate (written in SQL or a,notebook) → Fill missing (imputed from the full,column) → Train (). Serving · per request: Live request (partial data, nothing,settled) → Aggregate again (reimplemented in app code) → Fill missing (a default someone chose) → Predict (). The fix: One feature definition () → Called by both paths () → Tested on identical input (same row in, same value,out) → Log served features (compare against training,later). Training · batch Warehouse table complete, corrected, late-arriving rows settled Aggregate written in SQL or a notebook Fill missing imputed from the full column Train Serving · per request Live request partial data, nothing settled Aggregate again reimplemented in app code Fill missing a default someone chose Predict The fix One feature definition Called by both paths Tested on identical input same row in, same value out Log served features compare against training later online, one row, now Where the two paths drift apart The fix — shared, tested once
The model learns from the left path and serves from the right. Shaded boxes are where the two silently diverge. The join at the bottom is the fix: one implementation, used by both.

The four places skew enters#

Reimplementation. The aggregate is written in SQL for training and again in application code for serving. Two authors, two readings of the same intent, one subtle difference — a boundary condition, a rounding rule, a timezone.

Time. Training sees a settled warehouse: late-arriving rows have landed, corrections have been applied. Serving sees the last few seconds. A feature meaning "orders in the past 30 days" is computed over different data in each path, and neither is wrong.

Missing values. Training imputes from the full column — a mean, a median. Serving has one row and uses whatever default someone typed. The model was taught that missing means "about average" and is now told it means zero.

Categories. A value that appears in serving but never in training. Encoders handle this inconsistently, and frequently silently.

How to detect it#

You cannot see it in evaluation, because evaluation runs entirely on the training path. Two things find it:

Log the features actually served, with the prediction. Then compare their distribution against training. A drift in one feature and not the others is the signature.

Test both paths on the same input. Take a row, run it through the training computation and the serving computation, and assert the outputs match. This is a cheap test and it catches most of the class.

Why the fix is structural#

The shared definition at the bottom of the diagram is not a tidiness preference. As long as two implementations exist, they will drift — someone fixes a bug in one, adds a condition, changes a default. The drift is invisible until accuracy falls.

One definition, called by both, is the only version of this that stays correct without vigilance.

See machine learning for the wider failure modes and the model review checklist before shipping.

Back to Machine Learning