Data Lab: A Pipeline That Refuses to Publish

Anybody can build a dashboard. The demonstrable thing about data engineering is what the pipeline does when the data is wrong, and the honest answer in most organisations is that it publishes it.

The five gates, and why these five#

They are the ones that catch most of what actually goes wrong.

Row count against expectation. A partial load looks like a genuine business decline on every chart it reaches. An empty batch is worse, because an empty table renders as a chart with no bars rather than as an error.

Completeness. A null summed as zero is a number that is wrong rather than missing, and it looks entirely plausible.

Key uniqueness. Usually a retry that partially succeeded. Every total downstream is overstated and nothing about the data looks wrong.

Distribution. The shape of a decimal-point or units error. One row can dominate every average it enters.

Freshness. Yesterday's data arriving as if it were today's is the failure that survives longest, because everything renders perfectly.

Stop publication, do not raise a ticket#

This is the part that matters and the part most pipelines get wrong.

A gate that raises a ticket lets the bad number through and starts a race between your alerting and your readers. Your readers usually win. By the time anyone acts, the figure has been screenshotted into a deck.

A gate that blocks publication leaves yesterday's numbers in place and pages the owner. The dashboard is stale and correct, which is recoverable. The alternative is fresh and wrong, which costs the credibility of every other number on the surface.

Note what the demo does not do when a gate fails: it does not partially publish. There is no state where half the dashboard is today and half is yesterday, which is the condition that destroys trust fastest because it is the hardest to notice.

Gates need an expectation, not a rule#

Every check above compares against something: an expected row count, a known range, a prior date. That is why an instruction to "add data quality checks" so often produces nothing useful. Without a baseline there is nothing to compare against, and the check degrades into asserting that the file exists.

Establishing the baseline is most of the work and none of the glamour, which is why it is usually skipped.

What this costs#

Five gates run in seconds and are perhaps a day of engineering per pipeline. Set against one incident where a wrong number reached an executive, the comparison is not close.

We build them into the pipeline rather than beside it, because a check running somewhere else on its own schedule is a check that will eventually disagree with the pipeline it is supposed to be guarding.

What else is coming for Data Lab