Checklist · Data Engineering

Data Pipeline Checklist

Build a pipeline whose numbers can be trusted — raw layer, idempotency, the five quality checks that catch real problems, and the definitions that stop two dashboards disagreeing.

Download the checklist Read the full guide

Markdown. No sign-up, no email.

Pipeline: _______________ Date: _______ Owner: _______________

1. Definitions — before any code#

[ ] The metric this pipeline feeds is defined in writing
[ ] Exact calculation recorded
[ ] What is included and excluded, recorded
[ ] Authoritative source named — one system of record per domain
[ ] Definition owner named
[ ] Date the definition was agreed

Most "our dashboards disagree" problems are definition problems, not pipeline bugs.

2. Structure#

[ ] Raw data stored exactly as received, before any transformation
[ ] Transformations happen after loading (ELT), so a bug is fixable by re-running
[ ] Transformations in version control, reviewed like code
[ ] Business logic lives in the pipeline, not in the dashboard

Without a raw layer you cannot tell, six weeks later, whether the source was wrong or your transformation was.

3. Correctness properties#

[ ] Idempotent — running twice produces the same result, not double the rows
[ ] Re-runnable safely after a failure
[ ] Late-arriving data handled explicitly — does it update the original day or the arrival day?
[ ] History preserved where it matters — overwriting silently rewrites the past
[ ] Time zones handled consistently

4. The five checks that catch real problems#

[ ] Freshness — is the newest record as recent as it should be? Catches silent failure
[ ] Volume — is today's row count within the expected range?
[ ] Uniqueness — are keys that should be unique actually unique?
[ ] Nulls — is a normally-populated field suddenly empty?
[ ] Reconciliation — do totals match the system of record?

[ ] All five run automatically on every load
[ ] They fail loudly — a pipeline that fails silently is worse than one that does not run

5. Failure behaviour#

[ ] Fails closed — writes nothing rather than writing something wrong
[ ] Existing good data left intact on failure
[ ] Exits non-zero so the scheduler reports it
[ ] Writes are atomic — an interrupted run cannot leave a half-written file
[ ] Someone is alerted, and it is a person not a shared inbox

6. Test data and environments#

[ ] Test data resembles production in shape and volume, not just values
[ ] No real personal data in test environments
[ ] Repeatable — a test does not depend on the previous run's state

7. Operations#

[ ] Schedule documented, with the reason for the timing
[ ] Dependencies on other pipelines mapped
[ ] Backfill procedure written and tested
[ ] Retention policy set and enforced
[ ] Runbook for the common failures

8. Scale, honestly#

[ ] Volume measured, not assumed
[ ] Tooling matched to volume — millions of rows do not need streaming infrastructure
[ ] Real-time only where a decision genuinely changes on it

Quality check results#

Check	Threshold	Result
Freshness
Volume
Uniqueness
Nulls
Reconciliation

Sign-off#

	Name	Date
Built by
Definition owner (business)

Back to Data Engineering