ML model evaluation mistakes that make good numbers meaningless

The model scored 0.94 AUC offline. In production it is barely better than the rule it replaced.

Nothing broke. The evaluation was measuring something other than what you deployed against. These are the nine ways that happens, roughly in order of how often they show up.

1. Random splits on time-ordered data

The most expensive mistake in applied ML, and the easiest to make — train_test_split is the default and it shuffles.

# Wrong for anything with a time dimension
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

A random split puts March data in train and February data in test. The model trains on the future and is tested on the past, which is not the problem you will ever solve in production.

Random split                    Time-based split

train  ██░█░██░█░█░██░█         train  ████████████░░░░
test   ░░█░█░░█░█░█░░░█         test   ░░░░░░░░░░░░████
       shuffled through time            past --> future

from sklearn.model_selection import TimeSeriesSplit

df = df.sort_values("timestamp")
cutoff = df["timestamp"].quantile(0.8)
train, test = df[df.timestamp < cutoff], df[df.timestamp >= cutoff]

# For cross-validation, expanding windows — never shuffled folds
for tr_idx, te_idx in TimeSeriesSplit(n_splits=5).split(df):
    ...

This applies far beyond obvious time series. Churn prediction, fraud detection, recommendations, demand forecasting, credit scoring — anything where the world changes has a time dimension, whether or not the dataframe has a date column.

2. Accuracy on imbalanced data

# 0.3% of transactions are fraud
model.score(X_test, y_test)      # 0.997 — and completely worthless

Predicting "never fraud" scores 99.7%. Accuracy on imbalanced data measures the class balance, not the model.

from sklearn.metrics import (classification_report, confusion_matrix,
                             average_precision_score)

print(confusion_matrix(y_test, preds))
print(classification_report(y_test, preds, digits=3))

# PR-AUC, not ROC-AUC, when positives are rare
print("PR-AUC:", average_precision_score(y_test, probs))

Use PR-AUC rather than ROC-AUC when the positive class is rare. ROC-AUC's false-positive rate has the huge negative class in its denominator, so thousands of false positives barely move it. Precision has the predicted-positives in the denominator, so it reflects what an analyst reviewing alerts actually experiences.

100,000 transactions, 300 fraud

Model flags 1,000, catches 200 fraud, 800 false alarms
  ROC-AUC  : looks excellent (FPR = 800/99700 = 0.008)
  Precision: 200/1000 = 0.20  <- the analyst's reality

3. Reporting threshold-free metrics for a threshold-based decision

AUC summarises performance across all thresholds. You deploy exactly one.

# Reported: AUC 0.91.  Deployed: probs > 0.5.
# Nobody checked whether 0.5 is a sensible operating point.

Pick the threshold on validation data, using the cost structure of the actual decision:

import numpy as np
from sklearn.metrics import precision_recall_curve

prec, rec, thresholds = precision_recall_curve(y_val, probs_val)

# Example policy: the review team can handle 60% precision minimum
viable = prec[:-1] >= 0.60
best = np.argmax(rec[:-1] * viable)
threshold = thresholds[best]

print(f"threshold={threshold:.3f} precision={prec[best]:.3f} recall={rec[best]:.3f}")

Then report precision and recall at that threshold on the test set. AUC is a useful model-comparison number; it is not a deployment number.

4. Tuning on the test set

# Run 1: test accuracy 0.81 — try more trees
# Run 2: test accuracy 0.83 — try a different depth
# Run 3: test accuracy 0.86 — ship it

Every look at the test set leaks a little information through your decisions. Forty experiments later, the test score reflects how well you searched, not how well the model generalises. This is overfitting with the human as the optimiser, and it is invisible because no code is obviously wrong.

Three splits, and the third is opened once:

train, temp = train_test_split(df, test_size=0.3, shuffle=False)
val, test = train_test_split(temp, test_size=0.5, shuffle=False)

# All experiments, all hyperparameter search, all feature selection: val only.
# test: opened once, at the end, and the number is reported as-is.

If you must reuse the test set, treat the second look as a decision you have to defend, and expect the true generalisation gap to be wider than the number suggests.

5. Cross-validation with grouped data

One patient with 40 visits. One user with 300 sessions. One factory with 12 machines.

# Wrong: the same patient appears in train and validation
cross_val_score(model, X, y, cv=5)

The model memorises patient-specific quirks and is validated on the same patients. Scores look excellent, and the model fails on a new patient — which is the only case that matters.

from sklearn.model_selection import GroupKFold, StratifiedGroupKFold

cross_val_score(model, X, y, groups=df["patient_id"],
                cv=GroupKFold(n_splits=5))

# When classes are also imbalanced
cv = StratifiedGroupKFold(n_splits=5)

Ask what unit you are generalising to. New patients? Group by patient. New stores? Group by store. New time periods? Split by time. The split must mirror the deployment question.

6. Preprocessing before the split

# Wrong: the scaler saw the test set's mean and variance
X_scaled = StandardScaler().fit_transform(X)
X_train, X_test = train_test_split(X_scaled)

Subtle, common, and it inflates scores by a small amount that you will never notice. The same applies to imputation, feature selection, PCA, target encoding and resampling — anything that learns from data.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer

pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("select", SelectKBest(k=20)),
    ("model", GradientBoostingClassifier()),
])

# Every step now fits on training folds only — CV is honest by construction
cross_val_score(pipe, X_train, y_train, cv=5)

Use a Pipeline. It is not a style preference; it is the mechanism that makes the leak structurally impossible.

SMOTE deserves its own warning: resample inside the pipeline, on training folds only. Oversampling before the split copies minority examples into both sides and produces spectacular, meaningless scores.

7. No baseline

# "Our model achieves 0.87 accuracy"
# Predicting the majority class achieves 0.85

Without a baseline, a number is not evidence. Always compute at least three:

from sklearn.dummy import DummyClassifier

baselines = {
    "majority": DummyClassifier(strategy="most_frequent"),
    "stratified": DummyClassifier(strategy="stratified"),
}
for name, b in baselines.items():
    b.fit(X_train, y_train)
    print(name, b.score(X_test, y_test))

# And the one that matters most: whatever rule is in production today
print("current rule:", accuracy_score(y_test, existing_rule(X_test)))

The existing heuristic is the real baseline. A model that does not beat "flag transactions over £5,000 from new accounts" is not worth its maintenance cost, however good its AUC looks in isolation.

8. Single-number reporting with no uncertainty

# Model A: 0.847.  Model B: 0.851.  Ship B.

On a 2,000-row test set, that difference is noise. Report an interval:

import numpy as np


def bootstrap_ci(y_true, y_pred, metric, n=1000, alpha=0.05):
    rng = np.random.default_rng(0)
    scores = []
    for _ in range(n):
        idx = rng.integers(0, len(y_true), len(y_true))
        if len(np.unique(y_true[idx])) < 2:      # skip degenerate resamples
            continue
        scores.append(metric(y_true[idx], y_pred[idx]))
    lo, hi = np.percentile(scores, [100 * alpha / 2, 100 * (1 - alpha / 2)])
    return np.mean(scores), lo, hi


mean, lo, hi = bootstrap_ci(y_test, probs, roc_auc_score)
print(f"AUC {mean:.3f}  95% CI [{lo:.3f}, {hi:.3f}]")
# AUC 0.847  95% CI [0.821, 0.872]

Overlapping intervals mean you cannot distinguish the models on this data. That is a legitimate and useful conclusion — and it saves you from shipping a "better" model that is actually a coin flip.

9. Aggregate metrics that hide subgroup failure

One number over the whole test set averages away the failures that matter.

def by_segment(df, y_true, y_pred, column):
    rows = []
    for value, group in df.groupby(column):
        idx = group.index
        rows.append({
            column: value,
            "n": len(idx),
            "precision": precision_score(y_true[idx], y_pred[idx], zero_division=0),
            "recall": recall_score(y_true[idx], y_pred[idx], zero_division=0),
        })
    return pd.DataFrame(rows).sort_values("recall")


print(by_segment(test_df, y_test, preds, "region"))
print(by_segment(test_df, y_test, preds, "customer_tier"))
print(by_segment(test_df, y_test, preds, "signup_month"))

region     n      precision  recall
APAC       412    0.42       0.31     <- the model does not work here
EMEA      3891    0.79       0.74
US       11204    0.86       0.83
overall  15507    0.84       0.80     <- hides it completely

Slice by every dimension you can name: geography, customer segment, device, time period, data source, and any protected attribute relevant to fairness. Slice by data recency in particular — degrading performance on the most recent slice is drift arriving early, and it is the single most useful early warning you can build.

The checklist

Before you believe an evaluation:

• Split respects time, if time matters
• Split respects groups, if entities repeat
• Every preprocessing step lives inside a Pipeline
• Metric matches the decision — PR-AUC for rare positives, not accuracy
• Threshold chosen on validation, metrics reported at that threshold
• Test set opened once
• At least three baselines, including the current production rule
• Confidence intervals on every comparison
• Per-segment breakdown, including the most recent time slice

Frequently asked questions

Why is my model much worse in production than in testing?

The three usual causes, in order: a random split on time-ordered data, a leaked feature that is unavailable at prediction time, and preprocessing fitted before the split. Check the split first — it is the most common and the most damaging.

Should I use ROC-AUC or PR-AUC?

PR-AUC when the positive class is rare. ROC-AUC's false-positive rate is diluted by a huge negative class, so it stays flattering while precision collapses. If you would care about the ratio of true alerts to false alerts, use PR-AUC.

How many times can I look at the test set?

Once, ideally. Every look leaks information through the decisions you make afterwards. Use a validation set for all experimentation. If you must look again, treat the resulting number as optimistic and say so.

What is the right cross-validation for time series?

Expanding or rolling windows where training data always precedes validation data — TimeSeriesSplit in scikit-learn. Never shuffle. If you also have grouped entities, combine the two: split by time first, then check that groups do not straddle the boundary.

Is a 2% accuracy improvement significant?

Depends entirely on test set size and variance. Bootstrap a confidence interval; if the intervals overlap substantially, you cannot distinguish the models on this data. On a few thousand rows, differences under about 2 points are usually noise.

Next reading on Sythra Articles

• Data leakage: the kinds that survive your train/test split
• Production ML architecture
• How to detect overfitting in your own model