#spark

Data Engineering

Idempotent data pipelines: how to make every re-run safe

Daniel Tran · May 12, 2026 · 9 min read

A pipeline you are afraid to re-run is a pipeline that will eventually corrupt data. This guide covers partition overwrite, MERGE upserts, dedup keys, watermarks and why exactly-once is a property you build, not a setting you enable.

Performance & Scaling Bug Report

Spark batch scoring 20x slower than expected: the model was loaded inside every UDF call

Ryan Pham · Apr 15, 2026 · 5 min read

Scoring 80 million rows with a 600 MB model took over 9 hours on a 20-node Spark cluster. The pandas UDF loaded the model from storage on every Arrow batch, thousands of times per executor. An iterator UDF that loads once per task cut the job to 28 minutes.