How to detect overfitting in your own model

A student who memorises last year's exam paper scores brilliantly on that paper and fails the real one. That is overfitting.

Your model has learned the noise in your training data — the coincidences, the typos, the one weird row — instead of the pattern underneath. It looks excellent in your notebook and disappoints on anything new.

Here are three checks that reveal it, and the fixes in the order you should try them.

Check 1 — Compare training score to test score

The fastest test in machine learning:

train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)

print(f"Train: {train_score:.3f}")
print(f"Test:  {test_score:.3f}")
print(f"Gap:   {train_score - test_score:.3f}")

A perfect 1.000 training score is a warning, not an achievement.

Check 2 — Watch the validation curve

Plot performance against model complexity and you can see the exact point where learning turns into memorising:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import validation_curve
from sklearn.tree import DecisionTreeClassifier

depths = np.arange(1, 21)

train_scores, val_scores = validation_curve(
    DecisionTreeClassifier(random_state=42),
    X, y,
    param_name="max_depth",
    param_range=depths,
    cv=5,
)

plt.plot(depths, train_scores.mean(axis=1), label="train")
plt.plot(depths, val_scores.mean(axis=1), label="validation")
plt.xlabel("max_depth")
plt.ylabel("accuracy")
plt.legend()
plt.show()

The shape is always the same: both lines rise together, then the training line keeps climbing while the validation line flattens and turns down. The peak of the validation line is your setting. Everything to the right is memorisation.

Check 3 — Add data and see if it helps

A learning curve tells you whether more data would fix the problem — which saves you from collecting data that will not help:

from sklearn.model_selection import learning_curve

sizes, train_scores, val_scores = learning_curve(
    model, X, y,
    train_sizes=np.linspace(0.1, 1.0, 8),
    cv=5,
)

plt.plot(sizes, train_scores.mean(axis=1), label="train")
plt.plot(sizes, val_scores.mean(axis=1), label="validation")
plt.xlabel("training examples")
plt.legend()
plt.show()

Read it like this:

• Lines still converging at the right edge → more data will help
• Lines flat and far apart → more data will not help; simplify the model
• Lines together but both low → underfitting; use a stronger model

Fix 1 — Simplify the model

The most direct cure. Every model has complexity knobs:

# Decision tree — the classic overfitter
DecisionTreeClassifier(max_depth=5, min_samples_leaf=20)

# Random forest — more trees is fine, shallower trees help
RandomForestClassifier(n_estimators=300, max_depth=10, min_samples_leaf=5)

# Gradient boosting — lower rate, fewer rounds
GradientBoostingClassifier(learning_rate=0.05, n_estimators=100, max_depth=3)

An unrestricted decision tree will grow until every training row sits in its own leaf. That is a 100% training score and a lookup table, not a model.

Fix 2 — Add regularisation

RegularisationA penalty that discourages the model from relying too heavily on any one feature pushes the model toward simpler explanations:

from sklearn.linear_model import LogisticRegression, Ridge

# Smaller C = stronger penalty
LogisticRegression(C=0.1, max_iter=1000)

# Larger alpha = stronger penalty
Ridge(alpha=10.0)

The two flavours behave differently, and the difference is useful:

• L2 (Ridge) shrinks all coefficients toward zero — the safe default
• L1 (Lasso) drives some coefficients exactly to zero — doubles as feature selection

from sklearn.linear_model import LogisticRegression

sparse = LogisticRegression(penalty="l1", solver="liblinear", C=0.1)

Fix 3 — Cut the number of features

More columns than rows is a recipe for memorisation. Every extra feature is another chance to find a coincidence:

from sklearn.feature_selection import SelectKBest, f_classif

selector = SelectKBest(score_func=f_classif, k=10)
X_small = selector.fit_transform(X_train, y_train)

kept = X.columns[selector.get_support()]
print("Kept:", list(kept))

Put the selector inside a Pipeline so it is fitted on training folds only — otherwise selecting features from the full dataset is itself data leakage.

Fix 4 — Get more data (or make some)

More examples make coincidences harder to memorise. When you cannot collect more, augment what you have. For images:

import cv2
import numpy as np

def augment(image):
    if np.random.rand() > 0.5:
        image = cv2.flip(image, 1)

    angle = np.random.uniform(-12, 12)
    h, w = image.shape[:2]
    matrix = cv2.getRotationMatrix2D((w / 2, h / 2), angle, 1.0)
    image = cv2.warpAffine(image, matrix, (w, h))

    factor = np.random.uniform(0.85, 1.15)
    return np.clip(image * factor, 0, 255).astype(np.uint8)

Flips, small rotations and brightness shifts create new-but-valid examples. The model learns "a cat is a cat at any angle" rather than "a cat is those exact pixels".

Fix 5 — Stop training early

For iterative models, monitor validation performance and stop when it turns:

from sklearn.ensemble import GradientBoostingClassifier

model = GradientBoostingClassifier(
    n_estimators=1000,
    validation_fraction=0.15,
    n_iter_no_change=15,     # stop after 15 rounds with no improvement
    random_state=42,
)
model.fit(X_train, y_train)
print("Rounds used:", model.n_estimators_)

Early stoppingEnding training when validation performance stops improving is free regularisation — you get the best round instead of the last one.

The order to try things

1. Measure the train/test gap — know the size of the problem
2. Simplify the model (max_depth, min_samples_leaf)
3. Add regularisation (C, alpha)
4. Reduce features
5. Add or augment data
6. Use early stopping

Work top-down and stop when the gap closes. Most overfitting disappears at step 2.

FAQ

How do I know if my model is overfitting?

Compare the training score with the test score. A gap larger than about 0.15 — especially a near-perfect training score with a much lower test score — means the model has memorised the training data.

What is the difference between overfitting and underfitting?

Overfitting means high training score and low test score: the model learned noise. Underfitting means both scores are low: the model is too simple to capture the pattern at all.

How do I fix overfitting in a decision tree?

Limit its growth with max_depth and raise min_samples_leaf. An unrestricted tree keeps splitting until each training row has its own leaf, which is pure memorisation.

Does more data always fix overfitting?

No. Plot a learning curve — if the training and validation lines are flat and far apart, extra data will not close the gap and you should simplify the model instead.

What does regularisation actually do?

It adds a penalty for large coefficients, so the model prefers simpler explanations. L2 shrinks all weights toward zero; L1 sets some to exactly zero and effectively selects features.

Next reading on Sythra Articles

• Train/test split and the accuracy lie
• Confusion matrix, precision and recall explained
• Train your first ML model in 20 lines