# Data Pipeline Specification

> The question this document exists to answer: **when this pipeline produces nothing, or produces
> something wrong, who finds out and how?** Everything else is detail.

**Pipeline:** _______________
**Owner (person):** _______________
**Date:** _______  **Version:** _______

## 1. Purpose

**What decision or product depends on this data:** _______________

**Who consumes the output, and what they do with it:** _______________

**What happens downstream if it is a day late:** _______________

**What happens downstream if it is silently wrong:** _______________

> Those last two answers are usually very different, and they should drive different alerts.
> Late is visible. Wrong is not.

## 2. Source

| | |
|---|---|
| System | |
| Access method (API, file, database, feed) | |
| Owner of the source (person) | |
| Do they know we depend on it | yes / no |
| How we are told about schema changes | |
| Rate limits / quotas | |
| Historical availability, honestly | |

🔴 If nobody at the source knows you consume it, your pipeline will break the day they make a
routine change. Introduce yourself before that happens.

## 3. Contract with the source

| Field | Type | Nullable | Meaning | What we do if it is missing |
|---|---|---|---|---|
| | | | | |

**Expected volume per run:** _____ (min _____ / max _____)
**Expected freshness:** data should be no older than _____
**Known quirks:** _______________ (timezones, encodings, duplicate keys, retroactive edits)

## 4. Schedule and idempotency

| | |
|---|---|
| Runs | |
| Why that time (upstream availability?) | |
| Late-arriving data window | |
| **Safe to re-run the same period twice** | yes / no |
| How re-running is made safe | overwrite partition / upsert on key / other |
| Backfill procedure | |

A pipeline you cannot safely re-run cannot be recovered under pressure. If the answer above is
"no", fix that before anything else on this page.

## 5. Transformations

| Step | What it does | Assumption it relies on |
|---|---|---|
| | | |

**Business rules encoded here that are documented nowhere else:** _______________

## 6. Output

| | |
|---|---|
| Destination | |
| Write mode | append / overwrite / merge |
| Partitioning | |
| Schema owned by | |
| How consumers are told about changes | |
| Retention | |

## 7. Quality checks — the ones that catch silent failure

Run these on the OUTPUT, on every run, and fail the run when they fail.

| Check | Threshold | Action on failure |
|---|---|---|
| Row count within expected range | | fail / warn |
| **Row count not zero** | | **fail** |
| Freshness: max timestamp within ___ of now | | fail / warn |
| Primary key unique | | fail |
| Null rate per critical column below ___ | | fail / warn |
| Values within domain (currencies, statuses, ranges) | | fail / warn |
| Totals reconcile to source within ___ | | fail / warn |
| Distribution shift versus last ___ runs | | warn |

🔴 A job that succeeds having written zero rows is the most common silent data failure there is.
Exit codes describe whether code ran, not whether data arrived. Check the values.

## 8. Failure behaviour

| | |
|---|---|
| On source unavailable | retry ___ times, then ____ |
| **Does a failed run leave the previous good data intact** | yes / no |
| Partial write possible | yes / no — how prevented |
| Alert goes to | |
| Alert channel | |
| Alert fires on state change or every run | |
| Runbook location | |

**Fail closed:** when in doubt, keep the last good output and raise an alarm. Stale data that is
labelled stale is recoverable; empty or corrupted output propagates downstream in minutes.

## 9. Observability

- [ ] Run start, end, duration and row counts logged
- [ ] Metrics retained long enough to see a trend
- [ ] Someone would notice if this pipeline stopped running entirely
- [ ] Dashboard shows last successful run and freshness
- [ ] Cost per run known

**How would we notice if this pipeline stopped running at all?** _______________
(Monitoring the runs is not enough — a pipeline that is never triggered produces no failures.)

## 10. Sign-off

| | Name | Date |
|---|---|---|
| Built by | | |
| Source owner informed | | |
| Consumer accepted | | |
| In production since | | |