Data contracts: stop letting upstream deploys break your dashboards
A data contract is a schema, an owner and a test at the boundary. Here is the smallest version that actually prevents incidents.
The usual incident: an application team renames a column, nothing in their tests fails, and three hops downstream a revenue dashboard quietly reports zero. Support hears about it from a customer.
A data contract is an agreement between whoever produces a dataset and whoever depends on it. It does not need a platform to be useful. The smallest version has four parts.
The minimum viable contract
- A schema with types and nullability, stored in the producer’s repo.
- A named owner who gets paged when the contract is violated.
- Semantics: what a row means, its grain, and what “late” is.
- A check at the boundary that runs in the producer’s CI and again when data lands.
dataset: orders_events
owner: team-checkout
grain: one row per order state change
fields:
- name: order_id
type: string
required: true
- name: amount_cents
type: integer
required: true
freshness: 15m
Test where data crosses a team boundary
Tests deep inside a dbt project catch bugs late. Put the first check where the data enters your platform, and fail loudly there: reject or quarantine the batch, alert the producer, keep the bad rows out of the models downstream.
Lineage answers “who is affected?”
When a contract changes, you want the list of dependent models and dashboards in seconds, not a day of grepping. Column-level lineage from your orchestrator and dbt manifests is usually enough to start.
Start with the five tables that hurt
Don’t contract the whole warehouse. Pick the datasets behind your revenue reporting and customer-facing features, write the four parts above, and add the CI check. Expand when the first incident is prevented.
More from the blog
Batch to streaming without freezing your roadmap
A phased plan for moving nightly jobs onto streaming pipelines, with parallel runs and a cutover you can roll back.
Hire, freelancer, or dedicated pod: how to decide
Three ways for European companies to add data engineering capacity, what each really costs, and five questions that tell you which one fits.
Kafka consumer lag: the first alert to set up, and how to read it
Lag tells you whether consumers keep up. Alerting on its trend, not its size, catches stuck consumers before customers do.