--- title: "Production ML architecture: training-serving skew" description: "The model is the small part. Training-serving skew, feature freshness, shadow deploys, drift detection, and monitoring that catches decay." url: https://articles.sythra.ai/articles/production-ml-architecture slug: production-ml-architecture author: "vaibhavkothari" author_url: https://articles.sythra.ai/writers/vaibhavkothari date_published: 2026-08-29T09:26:55.771Z date_modified: 2026-08-29T10:22:15.966Z topics: ["Machine Learning", "Architecture", "Python", "Coding"] reading_time_minutes: 10 publisher: "Sythra" publisher_url: https://sythra.ai access: free language: en --- # Production ML architecture: training-serving skew > The model is the small part. Training-serving skew, feature freshness, shadow deploys, drift detection, and monitoring that catches decay. Source: https://articles.sythra.ai/articles/production-ml-architecture · Author: vaibhavkothari · Published: 2026-08-29 · Reading time: 10 min · Topics: Machine Learning, Architecture, Python, Coding A deployed model is not a deployed system. The model is perhaps 5% of the code and close to 0% of the operational risk. The other 95% is the machinery that gets features to the model, keeps offline and online agreeing, notices when the world moves, and lets you roll back. This article is about that machinery — specifically the parts that fail quietly. ## The shape ```text OFFLINE ONLINE raw data --> feature pipeline request --> feature lookup | | v v training data SAME transforms | | v v training --> model registry --------> inference | | | eval set | response v | monitoring <------------------+ | drift / decay alerts ``` The dangerous edge in that diagram is the one marked **SAME transforms**. Everything else is ordinary engineering. ## Training-serving skew The most common production failure, and it is silent. The model was trained on features computed one way and is served features computed another way. ```python # Training: pandas, batch, over the whole history df["orders_30d"] = (df.groupby("user_id")["order_id"] .rolling("30d").count().reset_index(0, drop=True)) # Serving: a hand-written SQL query in the API # SELECT COUNT(*) FROM orders WHERE user_id = ? AND created_at > NOW() - INTERVAL 30 DAY ``` Two implementations of "orders in the last 30 days". They disagree on: whether the boundary is inclusive, whether cancelled orders count, what timezone `NOW()` uses, and whether the training version included the current day. Every one of those is a silent distribution shift, and the model degrades without a single error being raised. **The structural fix is one implementation, used by both paths.** ```python from dataclasses import dataclass @dataclass(frozen=True) class FeatureSpec: name: str version: str def compute(self, events, as_of): """The single definition. Batch and online both call this.""" raise NotImplementedError class Orders30d(FeatureSpec): name, version = "orders_30d", "v2" def compute(self, events, as_of): window = events[ (events.created_at > as_of - pd.Timedelta(days=30)) & (events.created_at <= as_of) & (events.status != "cancelled") ] return len(window) ``` Batch training calls it with historical `as_of` values; the online path calls it with `now`. There is no second implementation to drift away from the first. The **version** field matters more than it looks. When the definition changes, the version changes, and models record which feature versions they were trained against. A model requesting `orders_30d:v1` from a store serving `v2` should fail loudly rather than quietly consuming a different number. ### Detecting skew you already have If you cannot restructure immediately, at least measure it. Log the features computed at serving time, and compare them against the batch pipeline's values for the same entities and timestamps. ```python def skew_report(online_log, batch_features, keys, tolerance=1e-6): merged = online_log.merge(batch_features, on=keys, suffixes=("_on", "_off")) rows = [] for col in FEATURE_COLUMNS: a, b = merged[f"{col}_on"], merged[f"{col}_off"] mismatch = (~np.isclose(a, b, rtol=tolerance, equal_nan=True)).mean() rows.append({"feature": col, "mismatch_rate": mismatch, "mean_abs_diff": (a - b).abs().mean()}) return pd.DataFrame(rows).sort_values("mismatch_rate", ascending=False) ``` Run it weekly. Any feature above a fraction of a percent mismatch is worth investigating; anything above 5% is actively degrading the model. ## Feature freshness, and its two failure modes A feature can be wrong by being stale or by being computed from incomplete data. ```text Batch pipeline runs at 02:00 daily. Request arrives at 23:00. Feature is 21 hours old. Was the model trained on features that were, on average, 21 hours old? If it was trained on same-instant features, this is skew. ``` The rule: **train on features with the same staleness distribution as production.** If serving uses a daily batch, training rows should use the value available at that day's batch time, not the exact value at the event instant. The second failure mode is worse because it looks like normal operation: ```python def get_features(user_id): features = store.get(user_id) if features is None: return DEFAULT_FEATURES # <- the quiet killer return features ``` When the pipeline fails, every request silently gets defaults and the model predicts the same thing for everyone. No error, no alert, degraded predictions for days. ```python def get_features(user_id, max_age=timedelta(hours=26)): row = store.get(user_id) if row is None: raise FeatureUnavailable(user_id) # fall back explicitly age = datetime.utcnow() - row.computed_at metrics.histogram("feature_age_seconds", age.total_seconds()) if age > max_age: raise FeatureStale(user_id, age) return row.features ``` Then the caller decides — fall back to a simpler model, to a heuristic, or return an explicit "no prediction". Whatever it does, it does it visibly. ## Deployment: shadow, then canary Never a straight swap. Two stages, and each answers a different question. **Shadow mode** answers "does the new model behave sanely on real traffic?" — it runs on live requests, logs its predictions, and its output is discarded. ```python async def predict(request): features = get_features(request.user_id) result = current_model.predict(features) if shadow_model is not None: # Never on the critical path: no await, no shared error handling asyncio.create_task(log_shadow(shadow_model, features, result, request)) return result async def log_shadow(model, features, live_result, request): try: shadow = model.predict(features) metrics.log({"request_id": request.id, "live": live_result, "shadow": shadow, "agreement": int(shadow.label == live_result.label)}) except Exception: metrics.increment("shadow_error") # never affects the response ``` Shadow mode catches feature incompatibility, latency regressions, crashes on real inputs, and unexpected prediction distributions — all without user impact. Run it for at least a full weekly cycle, because traffic on Sunday does not look like traffic on Tuesday. **Canary** answers "does it perform better?" — real traffic, a small share, measured on the actual business outcome. ```python def route(request): bucket = hash(f"{request.user_id}:{EXPERIMENT_SALT}") % 100 return new_model if bucket < CANARY_PERCENT else current_model ``` Hash on a **stable user identifier**, not on the request. A user who gets different models on successive requests produces incoherent experience and unusable experiment data. ## Monitoring: three layers, all necessary ### Layer 1 — the system Latency percentiles, error rate, throughput, feature-fetch failures, feature age. Standard service monitoring, and the only layer most teams actually have. ### Layer 2 — the predictions The model is unhealthy long before anyone complains. ```python def prediction_health(recent, reference): """Compare this hour's predictions against a known-good baseline.""" return { "mean_score": recent.score.mean(), "positive_rate": (recent.score > THRESHOLD).mean(), "psi": population_stability_index(recent.score, reference.score), "null_feature_rate": recent[FEATURE_COLUMNS].isna().mean().max(), } def population_stability_index(actual, expected, bins=10): """PSI > 0.2 is a meaningful distribution shift.""" edges = np.percentile(expected, np.linspace(0, 100, bins + 1)) edges[0], edges[-1] = -np.inf, np.inf a = np.histogram(actual, edges)[0] / len(actual) + 1e-6 e = np.histogram(expected, edges)[0] / len(expected) + 1e-6 return float(((a - e) * np.log(a / e)).sum()) ``` PSI conventions: below 0.1 is stable, 0.1–0.2 warrants a look, above 0.2 is a real shift. Track it on the prediction distribution **and** on each input feature — a shifted input feature tells you *which* upstream system changed, which is what you actually need at 3am. The single most informative alert here is a sudden change in **positive rate**. It moves before accuracy does, it needs no labels, and it catches upstream breakage almost immediately. ### Layer 3 — the outcome The only layer that measures whether the model is right, and the only one that is genuinely hard, because labels arrive late. ```text fraud : label in hours to days churn : label in 30-90 days credit risk : label in 6-24 months ``` ```python def delayed_evaluation(predictions, outcomes, lag_days): """Score only cohorts old enough to have labels.""" mature = predictions[ predictions.predicted_at < datetime.utcnow() - timedelta(days=lag_days) ] joined = mature.merge(outcomes, on="entity_id", how="left") # Rows still missing an outcome after the lag are negatives, not nulls — # get this wrong and your metrics are quietly optimistic. joined["label"] = joined["label"].fillna(0) return { "cohort": joined.predicted_at.dt.to_period("W"), "auc": roc_auc_score(joined.label, joined.score), "precision_at_threshold": precision_score( joined.label, joined.score > THRESHOLD), } ``` Report by **cohort week**, not overall. A single lifetime number hides a model that has been degrading for two months. ## Retraining, and how to decide Three triggers, and only one of them is a schedule: | Trigger | When it fires | Risk | |---|---|---| | **Scheduled** (weekly/monthly) | Always | Retrains when nothing changed; wastes cycles and adds churn | | **Drift-based** (PSI > threshold) | Input distribution moves | Fires on benign shifts too | | **Performance-based** (metric drops) | Outcome metrics degrade | Requires labels; slow for long-lag problems | In practice: scheduled retraining as the floor, drift alerts as an early warning that triggers a human decision, performance-based as the authority when labels exist. Whatever the trigger, the pipeline must be the same one that produced the current model, and the new model must clear the same gates: ```python def promote(candidate, champion, eval_set, holdout): checks = { "beats_champion": candidate.score(eval_set) > champion.score(eval_set), "no_segment_regression": all( candidate.score(s) >= champion.score(s) - 0.02 for s in segments(eval_set) ), "recent_slice_ok": candidate.score(holdout.last_month) >= MIN_SCORE, "latency_ok": candidate.p99_latency_ms < LATENCY_BUDGET, "feature_versions_match": candidate.feature_versions == store.versions(), } if not all(checks.values()): raise PromotionBlocked({k: v for k, v in checks.items() if not v}) return candidate ``` The `no_segment_regression` check is the one that saves you. An aggregate improvement that comes with a collapse on one region or customer tier is not an improvement — it is a trade you did not agree to make. ## What to log for every prediction You cannot debug what you did not record. The minimum: ```python log.info("prediction", extra={ "request_id": request.id, "entity_id": request.user_id, "model_name": model.name, "model_version": model.version, "feature_versions": model.feature_versions, "features": features.to_dict(), # the actual values used "feature_ages_s": feature_ages, "score": float(score), "threshold": THRESHOLD, "decision": decision, "latency_ms": elapsed_ms, }) ``` Logging **the feature values that were actually used** is what makes every later investigation possible: skew detection, drift attribution, and answering "why did the model do that on 14 March". Without it, every incident is archaeology. Sample if volume demands it — but sample the logs, never the fields. ```remember # Remember this Training-serving skew -> two implementations of one feature, drifting apart Stale features -> silent; missing features returning defaults are worse Shadow, then canary -> never a straight swap --- Log the feature values you actually used. Without them every incident becomes archaeology. ``` ## Frequently asked questions ### What is training-serving skew? When features are computed differently for training and for inference — different code, different filters, different timezones, different freshness. The model receives inputs from a distribution it was not trained on, and degrades silently. The fix is one feature implementation used by both paths. ### How often should I retrain? Start with a schedule matched to how fast your domain moves, then let drift and performance monitoring adjust it. Scheduled retraining alone either wastes effort or reacts too late; monitoring tells you which. Whatever the trigger, gate promotion on per-segment metrics, not only the aggregate. ### What is a feature store actually for? Two things: guaranteeing that training and serving use the same feature definitions, and supporting point-in-time correct historical lookups so training data reflects what was knowable at the time. If it is only a cache, it is solving the easy half. ### How do I detect model drift without labels? Monitor input feature distributions and the prediction distribution with PSI or a KS test against a reference window. Above about 0.2 PSI is a real shift. A sudden change in positive rate is the fastest label-free signal that something upstream broke. ### Should I deploy a new model straight to production? No. Shadow first — run it on real traffic with its output discarded — for at least one full weekly cycle, which catches crashes, latency regressions and feature incompatibility. Then canary a small share of traffic, routed by a stable user hash, and compare on the business outcome. ## Next reading on Sythra Articles - [ML model evaluation mistakes](https://articles.sythra.ai/articles/ml-model-evaluation-mistakes) - [Data leakage: the kinds that survive your split](https://articles.sythra.ai/articles/data-leakage-in-machine-learning) - [Agentic AI architecture](https://articles.sythra.ai/articles/agentic-ai-architecture)