---
title: "Train/test split explained — and why your accuracy is a lie"
description: "A 99% accurate model that fails in production usually has data leakage. Learn the four ways test data sneaks into training and how a scikit-learn Pipeline prevents all of them."
url: https://articles.sythra.ai/articles/train-test-split-data-leakage
slug: train-test-split-data-leakage
author: "vaibhavkothari"
author_url: https://articles.sythra.ai/writers/vaibhavkothari
date_published: 2026-08-26T08:34:22.039Z
date_modified: 2026-08-29T10:29:25.265Z
topics: ["Machine Learning", "Python", "Learning"]
reading_time_minutes: 6
publisher: "Sythra"
publisher_url: https://sythra.ai
access: free
language: en
---

# Train/test split explained — and why your accuracy is a lie

> A 99% accurate model that fails in production usually has data leakage. Learn the four ways test data sneaks into training and how a scikit-learn Pipeline prevents all of them.

Source: https://articles.sythra.ai/articles/train-test-split-data-leakage · Author: vaibhavkothari · Published: 2026-08-26 · Reading time: 6 min · Topics: Machine Learning, Python, Learning

Your model scores 99% on the test set. You deploy it. It performs like a coin flip.

This is the most common disappointment in machine learning, and the cause is nearly always the same: **data leakage**. Information from the test set found its way into training, so your score measured memory instead of skill.

This guide covers the four leaks that cause it, in the order beginners hit them.

## First, why we split at all

A model with enough capacity can memorise every row it sees. Scoring it on those same rows is like giving a student the exam they already have the answer key to.

```python
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
```

The test set is a stand-in for the future. Its whole value comes from being untouched. Every leak below is a way of touching it by accident.

## Leak 1 — Scaling before splitting

The most common one, and it looks completely innocent:

```python
# WRONG
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)          # sees every row, test included

X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)
```

`fit_transform` computes the mean and standard deviation of **all** the data. Those numbers carry information about the test rows into training. Small effect, real effect — and it inflates your score.

```python
# RIGHT
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)     # learn from training only
X_test = scaler.transform(X_test)           # apply, do not re-learn
```

The rule to memorise: **`fit` on training, `transform` on both**. It applies to scalers, imputers, encoders and vectorizers alike.

## Leak 2 — Filling missing values too early

Same mistake wearing a different coat:

```python
# WRONG — the median includes test rows
df["age"] = df["age"].fillna(df["age"].median())
```

Split first, compute the median from training, apply it to both. In practice, stop doing this by hand and let a Pipeline handle it:

```python
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier

pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("model", RandomForestClassifier(n_estimators=200, random_state=42)),
])

pipe.fit(X_train, y_train)
print(pipe.score(X_test, y_test))
```

A Pipeline makes leakage structurally impossible: every step fits on training data only, and `transform` runs on test data at predict time. This is the single highest-value habit in this article.

## Leak 3 — A column that contains the answer

Sometimes a feature is a disguised copy of the label:

- Predicting churn, with a `cancellation_date` column
- Predicting fraud, with an `investigation_opened` flag
- Predicting a disease, with `treatment_prescribed`

Each is only known **after** the thing you are predicting has happened. In production the column would be empty. In your CSV it is filled in, so the model learns to read it and scores brilliantly.

Two symptoms give it away:

```python
model.fit(X_train, y_train)
print(model.score(X_test, y_test))          # suspiciously near 1.0

import pandas as pd
print(pd.Series(model.feature_importances_, index=X.columns)
        .sort_values(ascending=False).head())
```

If one feature dominates every other and your accuracy is above 98% on a messy real-world problem, be suspicious rather than pleased. Ask of every column: *would I actually know this at the moment of prediction?*

## Leak 4 — Shuffling time series

For anything with a time order — prices, weather, demand — a random split lets the model train on Thursday and test on Wednesday. Predicting the past is easy and worthless.

```python
# WRONG for time series
X_train, X_test = train_test_split(X, test_size=0.2, shuffle=True)

# RIGHT — the test set is the future
split_at = int(len(df) * 0.8)
train_df, test_df = df.iloc[:split_at], df.iloc[split_at:]
```

For cross-validation, use the time-aware splitter:

```python
from sklearn.model_selection import TimeSeriesSplit

splitter = TimeSeriesSplit(n_splits=5)
for train_idx, test_idx in splitter.split(X):
    ...
```

Duplicate rows cause the same problem in a subtler way: the identical row lands in both sets, so the model has literally seen the test example. Run `df.drop_duplicates()` before splitting.

## A better estimate: cross-validation

A single split is one roll of the dice. On small datasets the score can swing several points depending on which rows landed where.

```python
from sklearn.model_selection import cross_val_score

scores = cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
print("Scores:", scores.round(3))
print("Mean:", scores.mean().round(3), "±", scores.std().round(3))
```

Cross-validation gives you a mean and a spread. The spread is the useful part: 0.86 ± 0.02 is a trustworthy result, 0.86 ± 0.11 means you do not really know your accuracy yet.

Note that `cross_val_score` re-fits the pipeline on each fold — which is exactly why passing a Pipeline rather than pre-scaled data matters here too.

## Three-way split: train, validation, test

Once you start tuning settings, you need a third set. Every time you look at a score and change something, you leak a little information into your decisions:

```python
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp
)
# 60% train, 20% validation, 20% test
```

Tune against validation. Touch the test set **once**, at the very end, to report a final number. If you tune against the test set, you have simply moved the leak into your own head.

## The leak checklist

Before trusting any score:

1. Did every `fit_transform` happen after the split?
2. Are preprocessing steps inside a Pipeline?
3. Could any feature only be known after the outcome?
4. Is the data time-ordered? Then split by time
5. Did you drop duplicate rows before splitting?
6. Does cross-validation agree with the single split?
7. Does the model beat `DummyClassifier` by a meaningful margin?

Six of the seven take one line each. They save weeks.

```remember
# Remember this
Split first -> then scale, impute, and select features
`fit` on train -> `transform` on both. Never `fit` on test.
Time-ordered data -> split by time, never shuffle
---
If the test set influenced training in any way, your accuracy is a lie.
```


## FAQ

### What is data leakage in machine learning?

Data leakage is when information from outside the training set — the test rows, or knowledge only available after the outcome — reaches the model during training. It produces high scores in testing and poor performance in production.

### Why is my model 99% accurate but useless in production?

Almost always leakage. Either a feature secretly encodes the answer, or preprocessing was fitted on the full dataset before the split, so the test score is not a fair measure of new data.

### Should I scale before or after train_test_split?

After. Fit the scaler on the training set only, then apply it to the test set with `transform`. A scikit-learn Pipeline enforces this automatically.

### How do I split time series data correctly?

Never shuffle. Use the earliest rows for training and the most recent for testing, or use `TimeSeriesSplit` for cross-validation, so the model is always predicting forwards in time.

### What is the difference between a validation set and a test set?

The validation set is for tuning — you may look at it repeatedly. The test set is used once, at the end, to report an honest final score.

## Next reading on Sythra Articles

- [Train your first ML model in 20 lines](https://articles.sythra.ai/articles/first-machine-learning-model-scikit-learn)
- [How to detect overfitting in your own model](https://articles.sythra.ai/articles/how-to-detect-overfitting)
- [Confusion matrix, precision and recall explained](https://articles.sythra.ai/articles/confusion-matrix-precision-recall-explained)

## Glossary (terms defined in this article)
- **Pipeline** (medium) — A scikit-learn object that chains preprocessing and a model into one unit
- **Cross-validation** (medium) — Splitting the data five ways and training five times so every row is tested once