Our daily signals disagreed with the exchange chart, and intraday signals appeared and disappeared within the same bar. We resampled UTC millisecond timestamps in local time and included the still-forming last candle. UTC everywhere and using only closed candles fixed both problems.
A churn model reached 0.97 AUC in validation and 0.68 in production. The training set joined customer features from the current snapshot, so it included support tickets and plan downgrades that happened after the prediction date. Point-in-time joins brought validation to an honest 0.81.
A moving-average strategy showed strong backtest returns and nothing like it in paper trading. The backtest computed the signal from a candle's close and filled the order at that same close, a price nobody could trade at once the signal was known. Shifting signals to the next bar fixed the simulation.