Cross-validation: why one train/test split lies

You split your data, trained a model, and got 97% accuracy. Is the model 97% good? Maybe. Or maybe you got a friendly split. A single train/test split is one roll of the dice, and a dice roll is not a measurement. Cross-validationEvaluating a model several times on different train/test splits and averaging the results turns it into one.

The problem: scores depend on the split

Run the same model on five different random splits of the breast-cancer dataset:

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

for seed in range(5):
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.25, random_state=seed
    )
    model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
    model.fit(X_train, y_train)
    print(seed, round(model.score(X_test, y_test), 3))

Typical output:

0 0.958
1 0.979
2 0.972
3 0.972
4 0.958

Same model, same data, scores from 95.8% to 97.9%. If you had only run seed 1, you would have reported 97.9% and felt great. With 569 rows the test set is just 143 examples, so a handful of tricky cases landing on one side of the split moves the score by whole percentage points.

On a small dataset the effect is much bigger. It is also why comparing two models on a single split is unreliable: a 0.5% difference may be nothing but split luck.

How k-fold cross-validation works

Instead of one split, divide the data into k equal folds (commonly 5). Then take turns:

Round 1: [TEST ][train][train][train][train]
Round 2: [train][TEST ][train][train][train]
Round 3: [train][train][TEST ][train][train]
Round 4: [train][train][train][TEST ][train]
Round 5: [train][train][train][train][TEST ]

Every example is used for testing exactly once and for training four times. You get five scores. Report their mean and spread.

In scikit-learn

from sklearn.model_selection import StratifiedKFold, cross_val_score

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(model, X, y, cv=cv)

print(scores.round(3))                       # [0.974 0.947 0.965 0.991 0.991]
print(f"{scores.mean():.3f} ± {scores.std():.3f}")   # 0.974 ± 0.017

Now the honest statement is: "about 97.4% accuracy, give or take 1.7 points." The ± is what a single split could never tell you. If two models differ by less than that spread, you cannot claim one is better.

Use stratified folds for classification

StratifiedKeeping the same proportion of each class in every fold splitting keeps the class balance consistent. Without it, a rare class (say 5% fraud) can end up almost absent from a fold, making that fold's score meaningless. For classifiers, pass StratifiedKFold — scikit-learn already does this by default when you give cv=5 and a classifier, but being explicit documents your intent.

For time-ordered data use TimeSeriesSplit, which always trains on the past and tests on the future. Shuffled folds on time series let the model peek at the future. For data with groups (several rows per patient or per user) use GroupKFold so one person never appears in both train and test.

The big trap: preprocessing outside the loop

Look again at the code above. The scaler lives inside the pipeline. That is not stylistic; it is what keeps the result honest.

The wrong way:

# DON'T: scaling uses information from the whole dataset, including test folds
X_scaled = StandardScaler().fit_transform(X)
scores = cross_val_score(LogisticRegression(max_iter=1000), X_scaled, y, cv=5)

Here the scaler computed its mean and standard deviation using rows that later act as the test fold. That is data leakage: information from the test data influenced training. For simple scaling the inflation is small. For feature selection, target encoding or imputation it can be dramatic — and the score looks better, so nobody notices.

The rule: anything that learns from data must be fitted only on the training portion of each fold. A Pipeline guarantees that, because cross_val_score refits the whole pipeline in every round.

Getting more than one metric

from sklearn.model_selection import cross_validate

results = cross_validate(
    model, X, y, cv=cv,
    scoring=["accuracy", "precision", "recall", "f1"],
    return_train_score=True,
)

for name in ["accuracy", "precision", "recall", "f1"]:
    vals = results[f"test_{name}"]
    print(f"{name:10s} {vals.mean():.3f} ± {vals.std():.3f}")

return_train_score=True lets you compare train and test per fold; a big gap signals overfitting. When classes are imbalanced, accuracy hides problems, so also look at the metrics in our confusion matrix guide.

Tuning hyperparameters: nested thinking

If you try ten settings with cross-validation and pick the best score, that winning score is slightly optimistic — you selected it because it scored well. For a final number you can trust, keep a separate test set that no tuning ever touched, or use nested cross-validation:

from sklearn.model_selection import GridSearchCV

inner = GridSearchCV(
    make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
    param_grid={"logisticregression__C": [0.01, 0.1, 1, 10]},
    cv=3,
)
outer_scores = cross_val_score(inner, X, y, cv=5)
print(outer_scores.mean())

The inner loop tunes; the outer loop measures how well the whole tuning procedure generalises.

When not to bother

• Huge datasets or slow models: one well-sized validation set may be enough, since a large test set is already stable. Five trainings of a model that takes a day is rarely worth it.
• Final deployment: after choosing a configuration, retrain on all the data, then ship that single model. Cross-validation is for estimating performance, not producing the model.

Checklist

1. Wrap preprocessing and the model in one Pipeline
2. Use StratifiedKFold (classification), GroupKFold (grouped rows) or TimeSeriesSplit (time)
3. Report mean and standard deviation
4. Use a held-out test set for the final number if you tuned anything
5. Retrain on all data for deployment

FAQ

What is cross-validation in machine learning?

Cross-validation evaluates a model on several different train/test splits and averages the scores. This gives a more reliable estimate of performance than a single split, plus a sense of how much the score varies.

How many folds should I use?

Five or ten is standard. More folds use more training data per round but cost more time. Five is a sensible default.

Is cross-validation better than train_test_split?

For estimating performance on small or medium data, yes — it is steadier and uses every example for testing. For very large data a single split is often fine.

Does cross-validation prevent data leakage?

Only if all learned preprocessing happens inside the cross-validation loop. Put scaling, imputation and feature selection in a Pipeline so they are refitted on each training fold.

Next reading on Sythra Articles

• Train/test split and data leakage
• Data leakage in machine learning
• Evaluation mistakes that make good numbers meaningless