Tag See all →

#etl

2 stories

Banner: a data pipeline that produces the same table no matter how many times it runs

Data Engineering

Idempotent data pipelines: how to make every re-run safe

Daniel Tran · May 12, 2026 · 9 min read

A pipeline you are afraid to re-run is a pipeline that will eventually corrupt data. This guide covers partition overwrite, MERGE upserts, dedup keys, watermarks and why exactly-once is a property you build, not a setting you enable.

Banner: Airflow DAG runs mapped to the wrong calendar days because of logical date confusion

Data Engineering Bug Report

Airflow daily DAG loaded the same day twice: logical date vs run time

Daniel Tran · Apr 24, 2025 · 10 min read

After a scheduler outage, our daily orders DAG caught up two missed runs and both loaded the same day while another day was skipped. The task computed its date from the wall clock instead of the run's data interval.