Checklist · Data Engineering

Data Pipeline Checklist

Build a pipeline whose numbers can be trusted, raw layer, idempotency, the five quality checks that catch real problems, and the definitions that stop two dashboards disagreeing.

Download the checklist (Word) Read the full guide

Markdown. No sign-up, no email.

Pipeline: _______________ Date: _______ Owner: _______________

1. Definitions: before any code#

[ ] The metric this pipeline feeds is defined in writing
[ ] Exact calculation recorded
[ ] What is included and excluded, recorded
[ ] Authoritative source named. One system of record per domain
[ ] Definition owner named
[ ] Date the definition was agreed

Most "our dashboards disagree" problems are definition problems, not pipeline bugs.

2. Structure#

[ ] Raw data stored exactly as received, before any transformation
[ ] Transformations happen after loading (ELT), so a bug is fixable by re-running
[ ] Transformations in version control, reviewed like code
[ ] Business logic lives in the pipeline, not in the dashboard

Without a raw layer you cannot tell, six weeks later, whether the source was wrong or your transformation was.

3. Correctness properties#

[ ] Idempotent. Running twice produces the same result, not double the rows
[ ] Re-runnable safely after a failure
[ ] Late-arriving data handled explicitly. Does it update the original day or the arrival day?
[ ] History preserved where it matters, overwriting silently rewrites the past
[ ] Time zones handled consistently

4. The five checks that catch real problems#

[ ] Freshness. Is the newest record as recent as it should be? Catches silent failure
[ ] Volume. Is today's row count within the expected range?
[ ] Uniqueness. Are keys that should be unique actually unique?
[ ] Nulls. Is a normally-populated field suddenly empty?
[ ] Reconciliation. Do totals match the system of record?

[ ] All five run automatically on every load
[ ] They fail loudly. A pipeline that fails silently is worse than one that does not run

5. Failure behaviour#

[ ] Fails closed. Writes nothing rather than writing something wrong
[ ] Existing good data left intact on failure
[ ] Exits non-zero so the scheduler reports it
[ ] Writes are atomic: an interrupted run cannot leave a half-written file
[ ] Someone is alerted, and it is a person not a shared inbox

6. Test data and environments#

[ ] Test data resembles production in shape and volume, not just values
[ ] No real personal data in test environments
[ ] Repeatable: a test does not depend on the previous run's state

7. Operations#

[ ] Schedule documented, with the reason for the timing
[ ] Dependencies on other pipelines mapped
[ ] Backfill procedure written and tested
[ ] Retention policy set and enforced
[ ] Runbook for the common failures

8. Scale, honestly#

[ ] Volume measured, not assumed
[ ] Tooling matched to volume: millions of rows do not need streaming infrastructure
[ ] Real-time only where a decision genuinely changes on it

Quality check results#

Check	Threshold	Result
Freshness
Volume
Uniqueness
Nulls
Reconciliation

Sign-off#

	Name	Date
Built by
Definition owner (business)

Back to Data Engineering