Churn model too good to be true: features joined as of today, not as of the label date

The retention team asked for a model that predicts, on the first day of each month, which subscribers will cancel during that month. The first version scored 0.97 AUC on a random validation split and 0.95 on a time-based split. Everyone was excited. In production, measured after one month of outcomes, AUC was 0.68.

Reproducing it

Feature importance was the first clue. The top three features were:

feature                          importance
support_tickets_last_30d           0.31
plan_downgraded_flag               0.22
days_since_last_login              0.14

For a customer who cancelled on March 18, we looked at the training row for the March 1 prediction:

customer_id  prediction_date  support_tickets_last_30d  plan_downgraded_flag  days_since_last_login
C-81233      2025-03-01       3                         1                     41

According to the ticket system, all three tickets were opened between March 10 and March 17. The downgrade happened on March 12. And "days since last login" was computed from the day the training set was built, in May.

What we ruled out

• Overfitting. A simpler logistic regression showed the same gap.
• A change in customer behaviour after launch. Recomputing the old validation period with correct features already gave much lower scores.

Root cause

The training query joined each label row (customer, prediction date, churned or not) to feature tables that held only the current state of each customer: customer_features was a snapshot rebuilt nightly. For historical prediction dates, the join pulled feature values computed after the prediction date, including events caused by the decision to churn (complaints, downgrades). In production, those events had not happened yet on the 1st of the month, so the model's strongest signals were missing. That is target leakage through time.

What is happening under the hood

Every training example represents a prediction made at a moment in time. Its features must contain only information available at that moment. With slowly changing data stored as current snapshots, it is easy to break that rule without noticing, because the join is syntactically fine.

A point-in-time correct join looks up, for each entity and timestamp, the latest feature value recorded at or before that timestamp. Feature stores implement this as "point-in-time joins"; in plain SQL it is an as-of join, typically with a window function or a lateral join.

Random train/validation splits hide the problem completely. Time-based splits help with other leakage, but not with features that are themselves computed in the future relative to each row.

Environment

Minimal reproduction

-- leaky: current snapshot joined to historical prediction dates
SELECT l.customer_id, l.prediction_date, l.churned, f.*
FROM labels l
JOIN customer_features f ON f.customer_id = l.customer_id;

The fix

1. Keep feature history. The nightly job now appends to customer_features_history with a computed_at timestamp instead of overwriting.

2. Build training sets with an as-of join.

SELECT l.customer_id, l.prediction_date, l.churned, f.*
FROM labels l
ASOF JOIN customer_features_history f
  MATCH_CONDITION (l.prediction_date >= f.computed_at)
  ON l.customer_id = f.customer_id;

For each label row, ASOF JOIN picks the latest feature row computed at or before the prediction date. Databases without as-of joins can do the same with a window function that ranks feature rows per label row and keeps the first.

For features computed directly from events, like ticket counts, the window ends at the prediction date:

SELECT l.customer_id, l.prediction_date,
       COUNT(e.id) AS support_tickets_last_30d
FROM labels l
LEFT JOIN events e
  ON e.customer_id = l.customer_id
 AND e.type = 'support_ticket'
 AND e.created_at >= DATEADD(day, -30, l.prediction_date)
 AND e.created_at <  l.prediction_date
GROUP BY 1, 2;

In pandas, merge_asof does the same for smaller datasets:

train = pd.merge_asof(
    labels.sort_values("prediction_date"),
    features.sort_values("computed_at"),
    left_on="prediction_date", right_on="computed_at",
    by="customer_id", direction="backward",
)

3. Backfill feature history for the past 18 months by replaying the feature computation over the event log with each historical date as "now", so we had training data without waiting.

Alternatives and trade-offs

• A feature store with built-in point-in-time joins (Feast, cloud provider feature stores). Removes a class of mistakes for shared features; more infrastructure to run.
• Log features at prediction time and train on logged features. The most faithful approach once a model is live; it does not help the first version.

How to test the fix

A leakage test asserts that for every training row, every source event used has a timestamp before the prediction date. We also added a canary: a deliberately leaky feature (the label shifted by one day) must be flagged by an automated check that warns when any single feature alone achieves AUC above 0.9.

Confirming in production

• Honest validation on a time-based split: AUC 0.81.
• Production AUC after two months of outcomes: 0.79, close to validation.
• The retention team adjusted their campaign targeting to the more realistic precision, which reduced wasted discount offers.

Regression guard

• The leakage test runs in the training pipeline.
• The single-feature AUC check runs on every new training set.
• Any feature table used for training must have history with timestamps; snapshot-only tables are rejected by the dataset builder.

Security implications

None directly. Feature history tables keep more personal data for longer than snapshot tables did, so they follow the same retention limits as the event data they come from (24 months) and are covered by deletion requests through customer id.

Prevention checklist

• Every training row has a prediction timestamp; features must be computed from data before it.
• Keep feature history, not only current snapshots.
• Use as-of joins or a feature store with point-in-time joins.
• Be suspicious of validation metrics that look too good.
• Check single-feature performance for leakage signals.

References

• Feast, point-in-time joins — https://docs.feast.dev/getting-started/concepts/point-in-time-joins
• pandas merge_asof — https://pandas.pydata.org/docs/reference/api/pandas.merge_asof.html
• scikit-learn, common pitfalls: data leakage — https://scikit-learn.org/stable/common_pitfalls.html#data-leakage
• Snowflake ASOF JOIN — https://docs.snowflake.com/en/sql-reference/constructs/asof-join