Tag See all →

#feature-engineering

2 stories

Banner: machine learning feature leakage from joining data that did not exist at prediction time

Data Engineering Bug Report

Churn model too good to be true: features joined as of today, not as of the label date

Ryan Pham · Jun 25, 2025 · 6 min read

A churn model reached 0.97 AUC in validation and 0.68 in production. The training set joined customer features from the current snapshot, so it included support tickets and plan downgrades that happened after the prediction date. Point-in-time joins brought validation to an honest 0.81.

Banner: training-serving skew from different preprocessing in training and production

AI & Machine Learning Bug Report

Model with 0.91 AUC offline performs near random online: training-serving skew

Ryan Pham · Nov 27, 2024 · 5 min read

Our fraud model scored 0.91 AUC on holdout data and close to random in production. The serving service re-implemented preprocessing in another language: the scaler was refit on each batch, one-hot categories came in a different order and text was not lowercased. Shipping the pipeline with the model fixed it.