ML model evaluation mistakes that make good numbers meaningless
Nine errors behind great offline metrics and bad production models: random splits on time series, threshold-free metrics, tuning on the test set.
Measuring models honestly, finding the leaks that survive a correct split, and keeping them working after deployment.
Nine errors behind great offline metrics and bad production models: random splits on time series, threshold-free metrics, tuning on the test set.
Splitting correctly stops the obvious leak. Target encoding, feature stores, label windows and proxy features leak anyway — how to find each.
The model is the small part. Training-serving skew, feature freshness, shadow deploys, drift detection, and monitoring that catches decay.
Accuracy hides the mistakes that matter. Learn to read a confusion matrix, tell precision from recall, and pick the metric that fits what your model is actually for.
Overfitting means your model memorised the training data instead of learning from it. Here are the three checks that reveal it and the fixes that work, in order.
Newsletter
Free essays on learning, Python, and ML — no account required. We’ll only email when there’s something worth reading.