Data Pipeline Architecture Diagram
The layered pipeline drawn with the raw layer that makes bugs fixable, the five quality checks that catch real problems, and the fail-closed behaviour that stops a broken run overwriting good data.
SVG. No sign-up, no email.
Every pipeline diagram shows data moving left to right. Very few show what happens when a step is wrong — which is the only thing that distinguishes a pipeline you can trust from one that merely runs.
Why the raw layer earns its place#
Land the data exactly as received, before any transformation. It is the single cheapest decision in the diagram and the one most often skipped.
Six weeks after a number goes wrong, there are two possible explanations: the source sent something different, or your transformation changed. Without a raw layer you cannot tell them apart, because the only surviving copy has already been transformed. With one, the question takes minutes — compare raw to output — and the fix is a re-run rather than a reconstruction.
It also changes what a bug costs. Transform-then-load means a transformation bug has destroyed the input. Load-then-transform means the input is still there, and every bug fixed today can be applied to all of history.
The five checks#
They are unglamorous and they catch nearly everything that matters.
Freshness catches the silent failure — the job that succeeds, writes nothing, and leaves last week's numbers on a dashboard nobody suspects. It is the most valuable of the five and the least often implemented.
Volume catches half-loads and duplicate loads. Today's row count against the expected range finds both.
Uniqueness catches the duplicate join, which does not error, does not look wrong, and quietly doubles a total.
Nulls catch upstream changes — a renamed column, a dropped field, a supplier who stopped sending something.
Reconciliation is the one that makes people believe the pipeline: does the total match the system of record? If it does not, everything downstream is a rounding error argument waiting to happen.
Run all five on every load, and make them fail loudly. A check that logs a warning nobody reads is a check that does not exist.
Failing closed#
The bottom lane is the behaviour that protects the numbers, and it is the opposite of what most pipelines do by default.
Write nothing rather than something wrong. A partial write is worse than no write, because it looks like a successful run.
Leave the last good output intact. Stale data is a visible problem — someone notices the date. Wrong data is an invisible one, and it is acted on.
Exit non-zero, or the scheduler records a success and nobody is told.
Alert a person, named, not a shared mailbox where responsibility evaporates.
Using this diagram#
Trace one pipeline you own against this and note the missing boxes. The common pattern is a pipeline with no raw layer, one check (usually volume), and a failure mode that writes an empty file over a good one. Fixing those three in that order takes about a day and removes the class of incident where the numbers were wrong for a fortnight before anyone noticed.