Train/test split explained — and why your accuracy is a lie

Your model scores 99% on the test set. You deploy it. It performs like a coin flip.

This is the most common disappointment in machine learning, and the cause is nearly always the same: data leakage. Information from the test set found its way into training, so your score measured memory instead of skill.

This guide covers the four leaks that cause it, in the order beginners hit them.

First, why we split at all

A model with enough capacity can memorise every row it sees. Scoring it on those same rows is like giving a student the exam they already have the answer key to.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

The test set is a stand-in for the future. Its whole value comes from being untouched. Every leak below is a way of touching it by accident.

Leak 1 — Scaling before splitting

The most common one, and it looks completely innocent:

# WRONG
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)          # sees every row, test included

X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)

fit_transform computes the mean and standard deviation of all the data. Those numbers carry information about the test rows into training. Small effect, real effect — and it inflates your score.

# RIGHT
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)     # learn from training only
X_test = scaler.transform(X_test)           # apply, do not re-learn

The rule to memorise: fit on training, transform on both. It applies to scalers, imputers, encoders and vectorizers alike.

Leak 2 — Filling missing values too early

Same mistake wearing a different coat:

# WRONG — the median includes test rows
df["age"] = df["age"].fillna(df["age"].median())

Split first, compute the median from training, apply it to both. In practice, stop doing this by hand and let a PipelineA scikit-learn object that chains preprocessing and a model into one unit handle it:

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier

pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("model", RandomForestClassifier(n_estimators=200, random_state=42)),
])

pipe.fit(X_train, y_train)
print(pipe.score(X_test, y_test))

A Pipeline makes leakage structurally impossible: every step fits on training data only, and transform runs on test data at predict time. This is the single highest-value habit in this article.

Leak 3 — A column that contains the answer

Sometimes a feature is a disguised copy of the label:

• Predicting churn, with a cancellation_date column
• Predicting fraud, with an investigation_opened flag
• Predicting a disease, with treatment_prescribed

Each is only known after the thing you are predicting has happened. In production the column would be empty. In your CSV it is filled in, so the model learns to read it and scores brilliantly.

Two symptoms give it away:

model.fit(X_train, y_train)
print(model.score(X_test, y_test))          # suspiciously near 1.0

import pandas as pd
print(pd.Series(model.feature_importances_, index=X.columns)
        .sort_values(ascending=False).head())

If one feature dominates every other and your accuracy is above 98% on a messy real-world problem, be suspicious rather than pleased. Ask of every column: would I actually know this at the moment of prediction?

Leak 4 — Shuffling time series

For anything with a time order — prices, weather, demand — a random split lets the model train on Thursday and test on Wednesday. Predicting the past is easy and worthless.

# WRONG for time series
X_train, X_test = train_test_split(X, test_size=0.2, shuffle=True)

# RIGHT — the test set is the future
split_at = int(len(df) * 0.8)
train_df, test_df = df.iloc[:split_at], df.iloc[split_at:]

For cross-validation, use the time-aware splitter:

from sklearn.model_selection import TimeSeriesSplit

splitter = TimeSeriesSplit(n_splits=5)
for train_idx, test_idx in splitter.split(X):
    ...

Duplicate rows cause the same problem in a subtler way: the identical row lands in both sets, so the model has literally seen the test example. Run df.drop_duplicates() before splitting.

A better estimate: cross-validation

A single split is one roll of the dice. On small datasets the score can swing several points depending on which rows landed where.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
print("Scores:", scores.round(3))
print("Mean:", scores.mean().round(3), "±", scores.std().round(3))

Cross-validationSplitting the data five ways and training five times so every row is tested once gives you a mean and a spread. The spread is the useful part: 0.86 ± 0.02 is a trustworthy result, 0.86 ± 0.11 means you do not really know your accuracy yet.

Note that cross_val_score re-fits the pipeline on each fold — which is exactly why passing a Pipeline rather than pre-scaled data matters here too.

Three-way split: train, validation, test

Once you start tuning settings, you need a third set. Every time you look at a score and change something, you leak a little information into your decisions:

X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp
)
# 60% train, 20% validation, 20% test

Tune against validation. Touch the test set once, at the very end, to report a final number. If you tune against the test set, you have simply moved the leak into your own head.

The leak checklist

Before trusting any score:

1. Did every fit_transform happen after the split?
2. Are preprocessing steps inside a Pipeline?
3. Could any feature only be known after the outcome?
4. Is the data time-ordered? Then split by time
5. Did you drop duplicate rows before splitting?
6. Does cross-validation agree with the single split?
7. Does the model beat DummyClassifier by a meaningful margin?

Six of the seven take one line each. They save weeks.

FAQ

What is data leakage in machine learning?

Data leakage is when information from outside the training set — the test rows, or knowledge only available after the outcome — reaches the model during training. It produces high scores in testing and poor performance in production.

Why is my model 99% accurate but useless in production?

Almost always leakage. Either a feature secretly encodes the answer, or preprocessing was fitted on the full dataset before the split, so the test score is not a fair measure of new data.

Should I scale before or after train_test_split?

After. Fit the scaler on the training set only, then apply it to the test set with transform. A scikit-learn Pipeline enforces this automatically.

How do I split time series data correctly?

Never shuffle. Use the earliest rows for training and the most recent for testing, or use TimeSeriesSplit for cross-validation, so the model is always predicting forwards in time.

What is the difference between a validation set and a test set?

The validation set is for tuning — you may look at it repeatedly. The test set is used once, at the end, to report an honest final score.

Next reading on Sythra Articles

• Train your first ML model in 20 lines
• How to detect overfitting in your own model
• Confusion matrix, precision and recall explained