--- title: "ML model evaluation mistakes that make good numbers meaningless" description: "Nine errors behind great offline metrics and bad production models: random splits on time series, threshold-free metrics, tuning on the test set." url: https://articles.sythra.ai/articles/ml-model-evaluation-mistakes slug: ml-model-evaluation-mistakes author: "vaibhavkothari" author_url: https://articles.sythra.ai/writers/vaibhavkothari date_published: 2026-08-29T09:26:55.771Z date_modified: 2026-08-29T10:22:15.966Z topics: ["Machine Learning", "Python", "Learning", "Coding"] reading_time_minutes: 9 publisher: "Sythra" publisher_url: https://sythra.ai access: free language: en --- # ML model evaluation mistakes that make good numbers meaningless > Nine errors behind great offline metrics and bad production models: random splits on time series, threshold-free metrics, tuning on the test set. Source: https://articles.sythra.ai/articles/ml-model-evaluation-mistakes · Author: vaibhavkothari · Published: 2026-08-29 · Reading time: 9 min · Topics: Machine Learning, Python, Learning, Coding The model scored 0.94 AUC offline. In production it is barely better than the rule it replaced. Nothing broke. The evaluation was measuring something other than what you deployed against. These are the nine ways that happens, roughly in order of how often they show up. ## 1. Random splits on time-ordered data The most expensive mistake in applied ML, and the easiest to make — `train_test_split` is the default and it shuffles. ```python # Wrong for anything with a time dimension X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) ``` A random split puts March data in train and February data in test. The model trains on the future and is tested on the past, which is not the problem you will ever solve in production. ```text Random split Time-based split train ██░█░██░█░█░██░█ train ████████████░░░░ test ░░█░█░░█░█░█░░░█ test ░░░░░░░░░░░░████ shuffled through time past --> future ``` ```python from sklearn.model_selection import TimeSeriesSplit df = df.sort_values("timestamp") cutoff = df["timestamp"].quantile(0.8) train, test = df[df.timestamp < cutoff], df[df.timestamp >= cutoff] # For cross-validation, expanding windows — never shuffled folds for tr_idx, te_idx in TimeSeriesSplit(n_splits=5).split(df): ... ``` This applies far beyond obvious time series. Churn prediction, fraud detection, recommendations, demand forecasting, credit scoring — anything where the world changes has a time dimension, whether or not the dataframe has a date column. ## 2. Accuracy on imbalanced data ```python # 0.3% of transactions are fraud model.score(X_test, y_test) # 0.997 — and completely worthless ``` Predicting "never fraud" scores 99.7%. Accuracy on imbalanced data measures the class balance, not the model. ```python from sklearn.metrics import (classification_report, confusion_matrix, average_precision_score) print(confusion_matrix(y_test, preds)) print(classification_report(y_test, preds, digits=3)) # PR-AUC, not ROC-AUC, when positives are rare print("PR-AUC:", average_precision_score(y_test, probs)) ``` **Use PR-AUC rather than ROC-AUC when the positive class is rare.** ROC-AUC's false-positive rate has the huge negative class in its denominator, so thousands of false positives barely move it. Precision has the predicted-positives in the denominator, so it reflects what an analyst reviewing alerts actually experiences. ```text 100,000 transactions, 300 fraud Model flags 1,000, catches 200 fraud, 800 false alarms ROC-AUC : looks excellent (FPR = 800/99700 = 0.008) Precision: 200/1000 = 0.20 <- the analyst's reality ``` ## 3. Reporting threshold-free metrics for a threshold-based decision AUC summarises performance across all thresholds. You deploy exactly one. ```python # Reported: AUC 0.91. Deployed: probs > 0.5. # Nobody checked whether 0.5 is a sensible operating point. ``` Pick the threshold on validation data, using the cost structure of the actual decision: ```python import numpy as np from sklearn.metrics import precision_recall_curve prec, rec, thresholds = precision_recall_curve(y_val, probs_val) # Example policy: the review team can handle 60% precision minimum viable = prec[:-1] >= 0.60 best = np.argmax(rec[:-1] * viable) threshold = thresholds[best] print(f"threshold={threshold:.3f} precision={prec[best]:.3f} recall={rec[best]:.3f}") ``` Then report precision and recall **at that threshold** on the test set. AUC is a useful model-comparison number; it is not a deployment number. ## 4. Tuning on the test set ```python # Run 1: test accuracy 0.81 — try more trees # Run 2: test accuracy 0.83 — try a different depth # Run 3: test accuracy 0.86 — ship it ``` Every look at the test set leaks a little information through your decisions. Forty experiments later, the test score reflects how well you searched, not how well the model generalises. This is overfitting with the human as the optimiser, and it is invisible because no code is obviously wrong. Three splits, and the third is opened once: ```python train, temp = train_test_split(df, test_size=0.3, shuffle=False) val, test = train_test_split(temp, test_size=0.5, shuffle=False) # All experiments, all hyperparameter search, all feature selection: val only. # test: opened once, at the end, and the number is reported as-is. ``` If you must reuse the test set, treat the second look as a decision you have to defend, and expect the true generalisation gap to be wider than the number suggests. ## 5. Cross-validation with grouped data One patient with 40 visits. One user with 300 sessions. One factory with 12 machines. ```python # Wrong: the same patient appears in train and validation cross_val_score(model, X, y, cv=5) ``` The model memorises patient-specific quirks and is validated on the same patients. Scores look excellent, and the model fails on a new patient — which is the only case that matters. ```python from sklearn.model_selection import GroupKFold, StratifiedGroupKFold cross_val_score(model, X, y, groups=df["patient_id"], cv=GroupKFold(n_splits=5)) # When classes are also imbalanced cv = StratifiedGroupKFold(n_splits=5) ``` **Ask what unit you are generalising to.** New patients? Group by patient. New stores? Group by store. New time periods? Split by time. The split must mirror the deployment question. ## 6. Preprocessing before the split ```python # Wrong: the scaler saw the test set's mean and variance X_scaled = StandardScaler().fit_transform(X) X_train, X_test = train_test_split(X_scaled) ``` Subtle, common, and it inflates scores by a small amount that you will never notice. The same applies to imputation, feature selection, PCA, target encoding and resampling — anything that learns from data. ```python from sklearn.pipeline import Pipeline from sklearn.impute import SimpleImputer pipe = Pipeline([ ("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler()), ("select", SelectKBest(k=20)), ("model", GradientBoostingClassifier()), ]) # Every step now fits on training folds only — CV is honest by construction cross_val_score(pipe, X_train, y_train, cv=5) ``` Use a `Pipeline`. It is not a style preference; it is the mechanism that makes the leak structurally impossible. **SMOTE deserves its own warning:** resample inside the pipeline, on training folds only. Oversampling before the split copies minority examples into both sides and produces spectacular, meaningless scores. ## 7. No baseline ```python # "Our model achieves 0.87 accuracy" # Predicting the majority class achieves 0.85 ``` Without a baseline, a number is not evidence. Always compute at least three: ```python from sklearn.dummy import DummyClassifier baselines = { "majority": DummyClassifier(strategy="most_frequent"), "stratified": DummyClassifier(strategy="stratified"), } for name, b in baselines.items(): b.fit(X_train, y_train) print(name, b.score(X_test, y_test)) # And the one that matters most: whatever rule is in production today print("current rule:", accuracy_score(y_test, existing_rule(X_test))) ``` The existing heuristic is the real baseline. A model that does not beat "flag transactions over £5,000 from new accounts" is not worth its maintenance cost, however good its AUC looks in isolation. ## 8. Single-number reporting with no uncertainty ```python # Model A: 0.847. Model B: 0.851. Ship B. ``` On a 2,000-row test set, that difference is noise. Report an interval: ```python import numpy as np def bootstrap_ci(y_true, y_pred, metric, n=1000, alpha=0.05): rng = np.random.default_rng(0) scores = [] for _ in range(n): idx = rng.integers(0, len(y_true), len(y_true)) if len(np.unique(y_true[idx])) < 2: # skip degenerate resamples continue scores.append(metric(y_true[idx], y_pred[idx])) lo, hi = np.percentile(scores, [100 * alpha / 2, 100 * (1 - alpha / 2)]) return np.mean(scores), lo, hi mean, lo, hi = bootstrap_ci(y_test, probs, roc_auc_score) print(f"AUC {mean:.3f} 95% CI [{lo:.3f}, {hi:.3f}]") # AUC 0.847 95% CI [0.821, 0.872] ``` Overlapping intervals mean you cannot distinguish the models on this data. That is a legitimate and useful conclusion — and it saves you from shipping a "better" model that is actually a coin flip. ## 9. Aggregate metrics that hide subgroup failure One number over the whole test set averages away the failures that matter. ```python def by_segment(df, y_true, y_pred, column): rows = [] for value, group in df.groupby(column): idx = group.index rows.append({ column: value, "n": len(idx), "precision": precision_score(y_true[idx], y_pred[idx], zero_division=0), "recall": recall_score(y_true[idx], y_pred[idx], zero_division=0), }) return pd.DataFrame(rows).sort_values("recall") print(by_segment(test_df, y_test, preds, "region")) print(by_segment(test_df, y_test, preds, "customer_tier")) print(by_segment(test_df, y_test, preds, "signup_month")) ``` ```text region n precision recall APAC 412 0.42 0.31 <- the model does not work here EMEA 3891 0.79 0.74 US 11204 0.86 0.83 overall 15507 0.84 0.80 <- hides it completely ``` Slice by every dimension you can name: geography, customer segment, device, time period, data source, and any protected attribute relevant to fairness. Slice by **data recency** in particular — degrading performance on the most recent slice is drift arriving early, and it is the single most useful early warning you can build. ## The checklist Before you believe an evaluation: - [ ] Split respects time, if time matters - [ ] Split respects groups, if entities repeat - [ ] Every preprocessing step lives inside a Pipeline - [ ] Metric matches the decision — PR-AUC for rare positives, not accuracy - [ ] Threshold chosen on validation, metrics reported at that threshold - [ ] Test set opened once - [ ] At least three baselines, including the current production rule - [ ] Confidence intervals on every comparison - [ ] Per-segment breakdown, including the most recent time slice ```remember # Remember this Random split on time-ordered data -> trains on the future Accuracy on imbalanced data -> measures the class balance AUC -> a model-comparison number, not a deployment number --- Pick the threshold on validation, report metrics at that threshold, per segment, with a confidence interval. Open the test set once. ``` ## Frequently asked questions ### Why is my model much worse in production than in testing? The three usual causes, in order: a random split on time-ordered data, a leaked feature that is unavailable at prediction time, and preprocessing fitted before the split. Check the split first — it is the most common and the most damaging. ### Should I use ROC-AUC or PR-AUC? PR-AUC when the positive class is rare. ROC-AUC's false-positive rate is diluted by a huge negative class, so it stays flattering while precision collapses. If you would care about the ratio of true alerts to false alerts, use PR-AUC. ### How many times can I look at the test set? Once, ideally. Every look leaks information through the decisions you make afterwards. Use a validation set for all experimentation. If you must look again, treat the resulting number as optimistic and say so. ### What is the right cross-validation for time series? Expanding or rolling windows where training data always precedes validation data — `TimeSeriesSplit` in scikit-learn. Never shuffle. If you also have grouped entities, combine the two: split by time first, then check that groups do not straddle the boundary. ### Is a 2% accuracy improvement significant? Depends entirely on test set size and variance. Bootstrap a confidence interval; if the intervals overlap substantially, you cannot distinguish the models on this data. On a few thousand rows, differences under about 2 points are usually noise. ## Next reading on Sythra Articles - [Data leakage: the kinds that survive your train/test split](https://articles.sythra.ai/articles/data-leakage-in-machine-learning) - [Production ML architecture](https://articles.sythra.ai/articles/production-ml-architecture) - [How to detect overfitting in your own model](https://articles.sythra.ai/articles/how-to-detect-overfitting)