# Data Pipeline Checklist

**Pipeline:** _______________  **Date:** _______  **Owner:** _______________

## 1. Definitions — before any code

- [ ] The metric this pipeline feeds is **defined in writing**
- [ ] Exact calculation recorded
- [ ] What is included and excluded, recorded
- [ ] **Authoritative source named** — one system of record per domain
- [ ] Definition owner named
- [ ] Date the definition was agreed

> Most "our dashboards disagree" problems are definition problems, not pipeline bugs.

## 2. Structure

- [ ] **Raw data stored exactly as received, before any transformation**
- [ ] Transformations happen after loading (ELT), so a bug is fixable by re-running
- [ ] Transformations in version control, reviewed like code
- [ ] Business logic lives in the pipeline, **not in the dashboard**

> Without a raw layer you cannot tell, six weeks later, whether the source was wrong or your
> transformation was.

## 3. Correctness properties

- [ ] **Idempotent** — running twice produces the same result, not double the rows
- [ ] Re-runnable safely after a failure
- [ ] **Late-arriving data handled explicitly** — does it update the original day or the arrival day?
- [ ] History preserved where it matters — overwriting silently rewrites the past
- [ ] Time zones handled consistently

## 4. The five checks that catch real problems

- [ ] **Freshness** — is the newest record as recent as it should be? *Catches silent failure*
- [ ] **Volume** — is today's row count within the expected range?
- [ ] **Uniqueness** — are keys that should be unique actually unique?
- [ ] **Nulls** — is a normally-populated field suddenly empty?
- [ ] **Reconciliation** — do totals match the system of record?

- [ ] All five run automatically on every load
- [ ] **They fail loudly** — a pipeline that fails silently is worse than one that does not run

## 5. Failure behaviour

- [ ] **Fails closed** — writes nothing rather than writing something wrong
- [ ] Existing good data left intact on failure
- [ ] Exits non-zero so the scheduler reports it
- [ ] Writes are atomic — an interrupted run cannot leave a half-written file
- [ ] Someone is alerted, and it is a person not a shared inbox

## 6. Test data and environments

- [ ] Test data resembles production in **shape and volume**, not just values
- [ ] No real personal data in test environments
- [ ] Repeatable — a test does not depend on the previous run's state

## 7. Operations

- [ ] Schedule documented, with the reason for the timing
- [ ] Dependencies on other pipelines mapped
- [ ] Backfill procedure written and tested
- [ ] Retention policy set and enforced
- [ ] Runbook for the common failures

## 8. Scale, honestly

- [ ] Volume measured, not assumed
- [ ] Tooling matched to volume — millions of rows do not need streaming infrastructure
- [ ] Real-time only where a decision genuinely changes on it

## Quality check results

| Check | Threshold | Result |
|---|---|---|
| Freshness | | |
| Volume | | |
| Uniqueness | | |
| Nulls | | |
| Reconciliation | | |

## Sign-off

| | Name | Date |
|---|---|---|
| Built by | | |
| Definition owner (business) | | |