---
title: "Cross-validation: why one train/test split lies"
description: "One train/test split gives one lucky or unlucky score. Learn k-fold and stratified cross-validation in scikit-learn, and how to avoid leaking data inside it."
url: https://articles.sythra.ai/articles/cross-validation-vs-single-split
slug: cross-validation-vs-single-split
author: "vaibhavkothari"
author_url: https://articles.sythra.ai/writers/vaibhavkothari
date_published: 2026-11-12T21:11:00.153Z
date_modified: 2026-11-12T21:11:00.153Z
topics: ["Machine Learning", "Python", "Scikit Learn"]
reading_time_minutes: 6
publisher: "Sythra"
publisher_url: https://sythra.ai
access: free
language: en
---

# Cross-validation: why one train/test split lies

> One train/test split gives one lucky or unlucky score. Learn k-fold and stratified cross-validation in scikit-learn, and how to avoid leaking data inside it.

Source: https://articles.sythra.ai/articles/cross-validation-vs-single-split · Author: vaibhavkothari · Published: 2026-11-12 · Reading time: 6 min · Topics: Machine Learning, Python, Scikit Learn

You split your data, trained a model, and got 97% accuracy. Is the model 97% good? Maybe. Or maybe you got a friendly split. A single train/test split is one roll of the dice, and a dice roll is not a measurement. Cross-validation turns it into one.

## The problem: scores depend on the split

Run the same model on five different random splits of the breast-cancer dataset:

```python
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

for seed in range(5):
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.25, random_state=seed
    )
    model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
    model.fit(X_train, y_train)
    print(seed, round(model.score(X_test, y_test), 3))
```

Typical output:

```
0 0.958
1 0.979
2 0.972
3 0.972
4 0.958
```

Same model, same data, scores from 95.8% to 97.9%. If you had only run seed 1, you would have reported 97.9% and felt great. With 569 rows the test set is just 143 examples, so a handful of tricky cases landing on one side of the split moves the score by whole percentage points.

On a small dataset the effect is much bigger. It is also why comparing two models on a single split is unreliable: a 0.5% difference may be nothing but split luck.

## How k-fold cross-validation works

Instead of one split, divide the data into `k` equal **folds** (commonly 5). Then take turns:

```
Round 1: [TEST ][train][train][train][train]
Round 2: [train][TEST ][train][train][train]
Round 3: [train][train][TEST ][train][train]
Round 4: [train][train][train][TEST ][train]
Round 5: [train][train][train][train][TEST ]
```

Every example is used for testing exactly once and for training four times. You get five scores. Report their **mean and spread**.

## In scikit-learn

```python
from sklearn.model_selection import StratifiedKFold, cross_val_score

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(model, X, y, cv=cv)

print(scores.round(3))                       # [0.974 0.947 0.965 0.991 0.991]
print(f"{scores.mean():.3f} ± {scores.std():.3f}")   # 0.974 ± 0.017
```

Now the honest statement is: *"about 97.4% accuracy, give or take 1.7 points."* The `±` is what a single split could never tell you. If two models differ by less than that spread, you cannot claim one is better.

## Use stratified folds for classification

Stratified splitting keeps the class balance consistent. Without it, a rare class (say 5% fraud) can end up almost absent from a fold, making that fold's score meaningless. For classifiers, pass `StratifiedKFold` — scikit-learn already does this by default when you give `cv=5` and a classifier, but being explicit documents your intent.

For time-ordered data use `TimeSeriesSplit`, which always trains on the past and tests on the future. Shuffled folds on time series let the model peek at the future. For data with groups (several rows per patient or per user) use `GroupKFold` so one person never appears in both train and test.

## The big trap: preprocessing outside the loop

Look again at the code above. The scaler lives **inside** the pipeline. That is not stylistic; it is what keeps the result honest.

The wrong way:

```python
# DON'T: scaling uses information from the whole dataset, including test folds
X_scaled = StandardScaler().fit_transform(X)
scores = cross_val_score(LogisticRegression(max_iter=1000), X_scaled, y, cv=5)
```

Here the scaler computed its mean and standard deviation using rows that later act as the test fold. That is [data leakage](https://articles.sythra.ai/articles/data-leakage-in-machine-learning): information from the test data influenced training. For simple scaling the inflation is small. For feature selection, target encoding or imputation it can be dramatic — and the score looks *better*, so nobody notices.

The rule: **anything that learns from data must be fitted only on the training portion of each fold.** A `Pipeline` guarantees that, because `cross_val_score` refits the whole pipeline in every round.

## Getting more than one metric

```python
from sklearn.model_selection import cross_validate

results = cross_validate(
    model, X, y, cv=cv,
    scoring=["accuracy", "precision", "recall", "f1"],
    return_train_score=True,
)

for name in ["accuracy", "precision", "recall", "f1"]:
    vals = results[f"test_{name}"]
    print(f"{name:10s} {vals.mean():.3f} ± {vals.std():.3f}")
```

`return_train_score=True` lets you compare train and test per fold; a big gap signals [overfitting](https://articles.sythra.ai/articles/how-to-detect-overfitting). When classes are imbalanced, accuracy hides problems, so also look at the metrics in our [confusion matrix guide](https://articles.sythra.ai/articles/confusion-matrix-precision-recall-explained).

## Tuning hyperparameters: nested thinking

If you try ten settings with cross-validation and pick the best score, that winning score is slightly optimistic — you selected it *because* it scored well. For a final number you can trust, keep a **separate test set** that no tuning ever touched, or use nested cross-validation:

```python
from sklearn.model_selection import GridSearchCV

inner = GridSearchCV(
    make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
    param_grid={"logisticregression__C": [0.01, 0.1, 1, 10]},
    cv=3,
)
outer_scores = cross_val_score(inner, X, y, cv=5)
print(outer_scores.mean())
```

The inner loop tunes; the outer loop measures how well the whole *tuning procedure* generalises.

## When not to bother

- **Huge datasets or slow models:** one well-sized validation set may be enough, since a large test set is already stable. Five trainings of a model that takes a day is rarely worth it.
- **Final deployment:** after choosing a configuration, retrain on **all** the data, then ship that single model. Cross-validation is for estimating performance, not producing the model.

## Checklist

1. Wrap preprocessing and the model in one `Pipeline`
2. Use `StratifiedKFold` (classification), `GroupKFold` (grouped rows) or `TimeSeriesSplit` (time)
3. Report mean **and** standard deviation
4. Use a held-out test set for the final number if you tuned anything
5. Retrain on all data for deployment

## FAQ

### What is cross-validation in machine learning?

Cross-validation evaluates a model on several different train/test splits and averages the scores. This gives a more reliable estimate of performance than a single split, plus a sense of how much the score varies.

### How many folds should I use?

Five or ten is standard. More folds use more training data per round but cost more time. Five is a sensible default.

### Is cross-validation better than train_test_split?

For estimating performance on small or medium data, yes — it is steadier and uses every example for testing. For very large data a single split is often fine.

### Does cross-validation prevent data leakage?

Only if all learned preprocessing happens inside the cross-validation loop. Put scaling, imputation and feature selection in a `Pipeline` so they are refitted on each training fold.

## Next reading on Sythra Articles

- [Train/test split and data leakage](https://articles.sythra.ai/articles/train-test-split-data-leakage)
- [Data leakage in machine learning](https://articles.sythra.ai/articles/data-leakage-in-machine-learning)
- [Evaluation mistakes that make good numbers meaningless](https://articles.sythra.ai/articles/ml-model-evaluation-mistakes)

## Glossary (terms defined in this article)
- **Cross-validation** (medium) — Evaluating a model several times on different train/test splits and averaging the results
- **Stratified** (medium) — Keeping the same proportion of each class in every fold