ETL, streaming, warehouses, data quality bugs.
A pipeline you are afraid to re-run is a pipeline that will eventually corrupt data. This guide covers partition overwrite, MERGE upserts, dedup keys, watermarks and why exactly-once is a property you build, not a setting you enable.
Scoring 80 million rows with a 600 MB model took over 9 hours on a 20-node Spark cluster. The pandas UDF loaded the model from storage on every Arrow batch, thousands of times per executor. An iterator UDF that loads once per task cut the job to 28 minutes.
Our support assistant quoted an old refund policy that had been removed from the help center weeks earlier. The ingestion pipeline only ever upserted chunks and never deleted them. We added source-of-truth sync, document versions and a nightly reconciliation job.
Our daily signals disagreed with the exchange chart, and intraday signals appeared and disappeared within the same bar. We resampled UTC millisecond timestamps in local time and included the still-forming last candle. UTC everywhere and using only closed candles fixed both problems.
A new operations dashboard refreshed every 15 minutes and our BigQuery bill jumped by 9,000 USD in a month. Each tile ran SELECT star over a 40 TB events table without a partition filter. Column pruning, required partition filters and a bytes billed cap fixed it.
Our perpetual futures bot reported a monthly PnL 6 percent higher than the exchange's account history. It ignored trading fees on some fills, never recorded funding payments, and valued open positions at last trade price instead of mark price. Building PnL from the income ledger fixed it.
A churn model reached 0.97 AUC in validation and 0.68 in production. The training set joined customer features from the current snapshot, so it included support tickets and plan downgrades that happened after the prediction date. Point-in-time joins brought validation to an honest 0.81.
Customer CSV uploads failed with KeyError 'email' even though the header clearly said email. Excel had written a UTF-8 byte order mark, and other files arrived in Windows-1252. We now detect the encoding explicitly and normalise headers.
After a scheduler outage, our daily orders DAG caught up two missed runs and both loaded the same day while another day was skipped. The task computed its date from the wall clock instead of the run's data interval.
A moving-average strategy showed strong backtest returns and nothing like it in paper trading. The backtest computed the signal from a candle's close and filled the order at that same close, a price nobody could trade at once the signal was known. Shifting signals to the next bar fixed the simulation.
Our enrichment consumers rebalanced every few minutes, lag climbed past two million messages and some records were processed twice. A slow external API pushed batch processing past max.poll.interval.ms. How we proved it, fixed the poll loop and made rebalances cheap.
Our nightly embedding backfill died on the same few documents every night. A handful of chunks were far over the embedding model's token limit because the chunker split by characters and some files were minified JSON. Token-aware chunking and batching fixed it.
Our local copy of the BTCUSDT order book slowly diverged from the exchange: crossed books, levels that never disappeared, and a best bid above the best ask. We applied depth diffs without buffering, skipped the snapshot alignment rules and never detected gaps. Following the sync procedure fixed it.