Data leakage: the kinds that survive your train/test split

You split before preprocessing. You used a pipeline. You split by time. And the model still scores 0.97 offline and 0.61 in production.

Data leakageInformation from outside the training-time reality reaching the model — usually the target, or the future, arriving through a feature does not only enter through the split. It enters through feature construction, through joins, through duplicated rows, and through features that are mathematically fine but temporally impossible. These are the kinds that survive a correct split.

The one-line detector

Before the taxonomy, the heuristic that catches most of it:

For every feature, ask: at the exact moment I need a prediction, does this value exist, and does it have this value?

Two clauses, and the second is the one that catches the subtle cases. account_status exists at prediction time — but the value in your training table is today's status, not the status as of the prediction date. That is leakage, and no split will save you.

1. Target encoding computed on the full dataset

The classic, and the most common leak in competition-style code.

# Wrong: every row's encoding includes its own target
df["city_encoded"] = df.groupby("city")["target"].transform("mean")

For a city with three rows, the encoding for row 1 is built from rows 1, 2 and 3 — including row 1's own label. The feature contains the answer. Worse, the leak is strongest for rare categories: a city with one row gets an encoding exactly equal to its target.

from sklearn.model_selection import KFold
import numpy as np


def oof_target_encode(df, col, target, n_splits=5, smoothing=20, seed=0):
    """Out-of-fold target encoding with smoothing toward the global mean."""
    global_mean = df[target].mean()
    encoded = np.full(len(df), np.nan)

    for tr, te in KFold(n_splits, shuffle=True, random_state=seed).split(df):
        stats = df.iloc[tr].groupby(col)[target].agg(["mean", "count"])
        # Shrink small categories toward the global mean
        smoothed = ((stats["mean"] * stats["count"] + global_mean * smoothing)
                    / (stats["count"] + smoothing))
        encoded[te] = df.iloc[te][col].map(smoothed).fillna(global_mean).values

    return encoded


df["city_encoded"] = oof_target_encode(df, "city", "target")

Two mechanisms are doing work here. Out-of-fold means a row's encoding never sees its own label. Smoothing shrinks small categories toward the global mean, so a category with two rows does not get a confident encoding built from two labels.

At inference, use the encoding fitted on the full training set — and store it, because recomputing it from production data is a fresh leak.

2. Feature stores serving current values for historical rows

The leak that survives everything, because the code is correct and the join is correct.

# The training table is built today
df = events.merge(customers, on="customer_id")     # customers = current state

events has a timestamp. customers does not — it holds today's lifetime_value, account_status, total_orders. A churn event from March gets joined to the customer's current state, which already reflects the churn.

event 2026-03-14: customer 4417 churned
join gives: account_status = "cancelled", lifetime_value = 890 (final)

The model learns "cancelled accounts churn". Perfect training accuracy.
At prediction time the status is "active" and the model has nothing.

The fix is a point-in-time join: every feature value must be as of the event timestamp.

# customer_history: (customer_id, valid_from, valid_to, status, ltv, ...)
features = pd.merge_asof(
    events.sort_values("event_time"),
    customer_history.sort_values("valid_from"),
    left_on="event_time",
    right_on="valid_from",
    by="customer_id",
    direction="backward",          # most recent state at or before the event
)

merge_asof with direction="backward" is the workhorse here. If your warehouse has no historical table, you cannot build honest features from it — that is a data engineering problem, and it is worth fixing before it is a modelling problem.

3. Label windows that overlap the feature window

You predict "will this customer churn in the next 30 days" using features from the last 90 days. If the feature window ends after the label window starts, features encode the outcome.

Wrong:
  features: [------- 90 days -------]
  label:              [--- 30 days ---]
                      ^^^^ overlap: features see the churn happening

Right:
  features: [------- 90 days -------]
                                     | prediction point
  label:                             [--- 30 days ---]

def build_row(customer_id, prediction_date, feature_days=90, label_days=30):
    feat_start = prediction_date - pd.Timedelta(days=feature_days)

    # Strictly before the prediction point
    window = events[
        (events.customer_id == customer_id)
        & (events.timestamp >= feat_start)
        & (events.timestamp < prediction_date)
    ]

    # Strictly after
    label_window = events[
        (events.customer_id == customer_id)
        & (events.timestamp >= prediction_date)
        & (events.timestamp < prediction_date + pd.Timedelta(days=label_days))
    ]

    return featurise(window), int((label_window.type == "churn").any())

Note the strict inequalities. <= on the feature side includes events at the prediction instant, which in practice often includes the event that is the label.

There is a second, sneakier version: the label is recorded before the outcome occurs. A fraud label added when an analyst opened the case, and a feature case_opened_at — the model learns to predict the investigation, not the fraud.

4. Duplicate and near-duplicate rows

Duplicates put the same information on both sides of the split, regardless of how you split.

print("exact duplicates:", df.duplicated().sum())
print("duplicate features, different label:",
      df.duplicated(subset=feature_cols).sum() - df.duplicated().sum())

Near-duplicates are worse because they survive duplicated(): the same support ticket submitted twice with different wording, the same product listed under two SKUs, augmented images of one original.

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np


def near_duplicate_pairs(texts, threshold=0.95, sample=5000):
    """Flag pairs that are almost certainly the same underlying item."""
    X = TfidfVectorizer(min_df=2).fit_transform(texts[:sample])
    S = (X @ X.T).toarray()
    np.fill_diagonal(S, 0)
    i, j = np.where(S > threshold)
    return [(a, b, S[a, b]) for a, b in zip(i, j) if a < b]

If duplicates are legitimate — the same entity genuinely appearing many times — do not delete them. Group by the entity and split by group, so all copies land on the same side.

5. Proxy features that encode the target's consequences

The feature does not contain the target, but it only exists because of it.

Predicting: will this loan default?
Feature:    number_of_collections_calls     <- only happens after default
Feature:    days_since_last_payment          <- fine, if computed as of the
                                                prediction date and not after
Predicting: will this patient be readmitted?
Feature:    discharge_medication_count       <- fine
Feature:    followup_appointment_booked      <- booked *because* of readmission risk

These are found by domain knowledge, not by code — but there is a reliable smell test: any single feature with implausibly high importance is a leak until proven otherwise.

import pandas as pd

imp = pd.Series(model.feature_importances_, index=feature_names)
print(imp.sort_values(ascending=False).head(10))

# Then: drop the top feature and retrain.
# If AUC falls from 0.97 to 0.71, that feature was doing all the work.
# Ask why, and be suspicious of the answer "it's just very predictive".

A single feature carrying 60% of the importance in a problem domain experts consider hard is almost always leakage.

6. Normalisation and imputation statistics from the whole dataset

Covered by pipelines, but worth naming because it survives a correct split when done manually:

# Wrong — median computed over train and test together
df["income"] = df["income"].fillna(df["income"].median())

# Right
median = train["income"].median()
train["income"] = train["income"].fillna(median)
test["income"] = test["income"].fillna(median)      # training statistic

Small effect, and it is the reason a model that looked reproducible offline shifts slightly when retrained. Use a Pipeline and it cannot happen.

7. Leakage through the identifier

Sometimes the ID itself carries the answer.

# Records were loaded in batches: fraud cases imported after clean ones
df["row_id"].corr(df["target"])       # 0.73

Auto-increment IDs, filenames with timestamps, and file order all encode collection history. If cases were gathered or labelled in a different pass from controls, the ID is a label.

Drop IDs from features by default, and check the correlation before you do.

Finding leakage you have not thought of

Three diagnostics, in increasing order of effort.

Too-good-to-be-true check. If the model beats what domain experts believe possible, it is leakage until proven otherwise. This is the most reliable signal in the whole article and it requires no code.

Adversarial validation. Train a classifier to distinguish train rows from test rows. It should fail.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

combined = pd.concat([train[features].assign(is_test=0),
                      test[features].assign(is_test=1)])

auc = cross_val_score(RandomForestClassifier(n_estimators=200),
                      combined[features], combined["is_test"],
                      cv=5, scoring="roc_auc").mean()
print(f"train/test discriminability: {auc:.3f}")
# ~0.5 = indistinguishable, healthy
# > 0.8 = the splits differ systematically — find out which feature and why

An AUC near 0.5 means train and test are drawn from the same distribution. A high AUC on a time-based split is expected to some degree — the world changed — but inspect the feature importances, because that is exactly where you will find the drifting or leaking column.

Ablation. Drop each feature group in turn and retrain. A group whose removal collapses performance deserves an explanation.

base = evaluate(model, X_val, y_val)
for group, cols in FEATURE_GROUPS.items():
    score = evaluate(refit(X_train.drop(columns=cols)), X_val.drop(columns=cols), y_val)
    print(f"{group:24s} without: {score:.3f}  (delta {score - base:+.3f})")

The audit, before any model ships

• Every feature is computable at prediction time, with that value
• All aggregates and encodings are computed out-of-fold or on training data only
• Joins to entity tables are point-in-time, not current-state
• Feature window ends strictly before the label window starts
• Duplicates and near-duplicates found, and grouped rather than split
• Top features inspected and explainable by a domain expert
• IDs and row order excluded from features
• Adversarial validation AUC is near 0.5, or the reason is understood
• Performance is plausible to someone who knows the domain

That last box is the one that catches what the others miss.

Frequently asked questions

What is data leakage in simple terms?

A feature contains information that would not be available — or would not have that value — at the moment you actually need a prediction. The model learns to use it, scores brilliantly offline, and has nothing to use in production.

How do I know if my model has data leakage?

The strongest signal is performance that is implausibly good for the problem. After that: one feature carrying most of the importance, a large gap between offline and production metrics, and a high adversarial-validation AUC between your train and test sets.

Is target encoding always leakage?

Only when it is computed on the full dataset. Out-of-fold target encoding with smoothing is a legitimate and effective technique — each row's encoding is built from folds that exclude it, and small categories are shrunk toward the global mean.

Why does splitting by time not stop all leakage?

Because leakage also enters through feature construction. Joining historical events to a current-state customer table leaks the future even with a perfect time split, since the joined values describe today, not the event date. Point-in-time joins are the fix.

What is adversarial validation?

Training a classifier to tell train rows from test rows. If it succeeds — AUC well above 0.5 — the two sets differ systematically, and the features driving that separation are where leakage or drift lives.

Next reading on Sythra Articles

• ML model evaluation mistakes
• Production ML architecture
• Train/test split and the leakage beginners hit first