Batch to streaming without freezing your roadmap

Migrations stall when they are framed as a rewrite. A quarter passes, the old system still runs, the new one isn’t trusted, and product work waits on both. The way out is to migrate one data flow at a time and keep the old path alive until the new one has earned trust.

Step 1: pick flows by pain, not by size

List your pipelines with three columns: how often they break, how stale their output is for the people using it, and what they cost to run. Start with one that is painful and self-contained, not the biggest one.

Step 2: land raw events first

Before changing any transformation, capture the source as an append-only event stream. Now you have a replayable record, and the old batch job and the new streaming job can both read it.

Step 3: run both, compare, then cut over

Run old and new in parallel for long enough to cover a month-end, a deploy and a bad day. Compare outputs automatically, table by table, and write down the accepted differences. A reconciliation report is what makes the cutover a decision instead of a leap.

Step 4: cut over with a way back

Switch consumers one at a time, keep the old job runnable for a few weeks, and define the rollback in advance: who decides, and what the trigger is.

Don’t forget cost

Streaming isn’t automatically cheaper. Track cost per pipeline before and after, and keep batch where hourly freshness is genuinely enough. Plenty of flows should stay in Airflow and dbt.

The aim is not “everything real-time”. It is that each flow runs at the freshness its users actually need, and that the team can explain why.