Production ML architecture: training-serving skew

A deployed model is not a deployed system. The model is perhaps 5% of the code and close to 0% of the operational risk.

The other 95% is the machinery that gets features to the model, keeps offline and online agreeing, notices when the world moves, and lets you roll back. This article is about that machinery — specifically the parts that fail quietly.

The shape

        OFFLINE                              ONLINE

  raw data --> feature pipeline          request --> feature lookup
                    |                                     |
                    v                                     v
              training data                        SAME transforms
                    |                                     |
                    v                                     v
                training  --> model registry --------> inference
                    |               |                     |
                 eval set           |                  response
                                    v                     |
                            monitoring <------------------+
                                    |
                            drift / decay alerts

The dangerous edge in that diagram is the one marked SAME transforms. Everything else is ordinary engineering.

Training-serving skew

The most common production failure, and it is silent. The model was trained on features computed one way and is served features computed another way.

# Training: pandas, batch, over the whole history
df["orders_30d"] = (df.groupby("user_id")["order_id"]
                      .rolling("30d").count().reset_index(0, drop=True))

# Serving: a hand-written SQL query in the API
# SELECT COUNT(*) FROM orders WHERE user_id = ? AND created_at > NOW() - INTERVAL 30 DAY

Two implementations of "orders in the last 30 days". They disagree on: whether the boundary is inclusive, whether cancelled orders count, what timezone NOW() uses, and whether the training version included the current day. Every one of those is a silent distribution shift, and the model degrades without a single error being raised.

The structural fix is one implementation, used by both paths.

from dataclasses import dataclass


@dataclass(frozen=True)
class FeatureSpec:
    name: str
    version: str

    def compute(self, events, as_of):
        """The single definition. Batch and online both call this."""
        raise NotImplementedError


class Orders30d(FeatureSpec):
    name, version = "orders_30d", "v2"

    def compute(self, events, as_of):
        window = events[
            (events.created_at > as_of - pd.Timedelta(days=30))
            & (events.created_at <= as_of)
            & (events.status != "cancelled")
        ]
        return len(window)

Batch training calls it with historical as_of values; the online path calls it with now. There is no second implementation to drift away from the first.

The version field matters more than it looks. When the definition changes, the version changes, and models record which feature versions they were trained against. A model requesting orders_30d:v1 from a store serving v2 should fail loudly rather than quietly consuming a different number.

Detecting skew you already have

If you cannot restructure immediately, at least measure it. Log the features computed at serving time, and compare them against the batch pipeline's values for the same entities and timestamps.

def skew_report(online_log, batch_features, keys, tolerance=1e-6):
    merged = online_log.merge(batch_features, on=keys, suffixes=("_on", "_off"))
    rows = []
    for col in FEATURE_COLUMNS:
        a, b = merged[f"{col}_on"], merged[f"{col}_off"]
        mismatch = (~np.isclose(a, b, rtol=tolerance, equal_nan=True)).mean()
        rows.append({"feature": col, "mismatch_rate": mismatch,
                     "mean_abs_diff": (a - b).abs().mean()})
    return pd.DataFrame(rows).sort_values("mismatch_rate", ascending=False)

Run it weekly. Any feature above a fraction of a percent mismatch is worth investigating; anything above 5% is actively degrading the model.

Feature freshness, and its two failure modes

A feature can be wrong by being stale or by being computed from incomplete data.

Batch pipeline runs at 02:00 daily.
Request arrives at 23:00.
Feature is 21 hours old.

Was the model trained on features that were, on average, 21 hours old?
If it was trained on same-instant features, this is skew.

The rule: train on features with the same staleness distribution as production. If serving uses a daily batch, training rows should use the value available at that day's batch time, not the exact value at the event instant.

The second failure mode is worse because it looks like normal operation:

def get_features(user_id):
    features = store.get(user_id)
    if features is None:
        return DEFAULT_FEATURES          # <- the quiet killer
    return features

When the pipeline fails, every request silently gets defaults and the model predicts the same thing for everyone. No error, no alert, degraded predictions for days.

def get_features(user_id, max_age=timedelta(hours=26)):
    row = store.get(user_id)
    if row is None:
        raise FeatureUnavailable(user_id)               # fall back explicitly

    age = datetime.utcnow() - row.computed_at
    metrics.histogram("feature_age_seconds", age.total_seconds())
    if age > max_age:
        raise FeatureStale(user_id, age)

    return row.features

Then the caller decides — fall back to a simpler model, to a heuristic, or return an explicit "no prediction". Whatever it does, it does it visibly.

Deployment: shadow, then canary

Never a straight swap. Two stages, and each answers a different question.

Shadow mode answers "does the new model behave sanely on real traffic?" — it runs on live requests, logs its predictions, and its output is discarded.

async def predict(request):
    features = get_features(request.user_id)
    result = current_model.predict(features)

    if shadow_model is not None:
        # Never on the critical path: no await, no shared error handling
        asyncio.create_task(log_shadow(shadow_model, features, result, request))

    return result


async def log_shadow(model, features, live_result, request):
    try:
        shadow = model.predict(features)
        metrics.log({"request_id": request.id,
                     "live": live_result, "shadow": shadow,
                     "agreement": int(shadow.label == live_result.label)})
    except Exception:
        metrics.increment("shadow_error")     # never affects the response

Shadow mode catches feature incompatibility, latency regressions, crashes on real inputs, and unexpected prediction distributions — all without user impact. Run it for at least a full weekly cycle, because traffic on Sunday does not look like traffic on Tuesday.

Canary answers "does it perform better?" — real traffic, a small share, measured on the actual business outcome.

def route(request):
    bucket = hash(f"{request.user_id}:{EXPERIMENT_SALT}") % 100
    return new_model if bucket < CANARY_PERCENT else current_model

Hash on a stable user identifier, not on the request. A user who gets different models on successive requests produces incoherent experience and unusable experiment data.

Monitoring: three layers, all necessary

Layer 1 — the system

Latency percentiles, error rate, throughput, feature-fetch failures, feature age. Standard service monitoring, and the only layer most teams actually have.

Layer 2 — the predictions

The model is unhealthy long before anyone complains.

def prediction_health(recent, reference):
    """Compare this hour's predictions against a known-good baseline."""
    return {
        "mean_score": recent.score.mean(),
        "positive_rate": (recent.score > THRESHOLD).mean(),
        "psi": population_stability_index(recent.score, reference.score),
        "null_feature_rate": recent[FEATURE_COLUMNS].isna().mean().max(),
    }


def population_stability_index(actual, expected, bins=10):
    """PSI > 0.2 is a meaningful distribution shift."""
    edges = np.percentile(expected, np.linspace(0, 100, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    a = np.histogram(actual, edges)[0] / len(actual) + 1e-6
    e = np.histogram(expected, edges)[0] / len(expected) + 1e-6
    return float(((a - e) * np.log(a / e)).sum())

PSI conventions: below 0.1 is stable, 0.1–0.2 warrants a look, above 0.2 is a real shift. Track it on the prediction distribution and on each input feature — a shifted input feature tells you which upstream system changed, which is what you actually need at 3am.

The single most informative alert here is a sudden change in positive rate. It moves before accuracy does, it needs no labels, and it catches upstream breakage almost immediately.

Layer 3 — the outcome

The only layer that measures whether the model is right, and the only one that is genuinely hard, because labels arrive late.

fraud       : label in hours to days
churn       : label in 30-90 days
credit risk : label in 6-24 months

def delayed_evaluation(predictions, outcomes, lag_days):
    """Score only cohorts old enough to have labels."""
    mature = predictions[
        predictions.predicted_at < datetime.utcnow() - timedelta(days=lag_days)
    ]
    joined = mature.merge(outcomes, on="entity_id", how="left")

    # Rows still missing an outcome after the lag are negatives, not nulls —
    # get this wrong and your metrics are quietly optimistic.
    joined["label"] = joined["label"].fillna(0)

    return {
        "cohort": joined.predicted_at.dt.to_period("W"),
        "auc": roc_auc_score(joined.label, joined.score),
        "precision_at_threshold": precision_score(
            joined.label, joined.score > THRESHOLD),
    }

Report by cohort week, not overall. A single lifetime number hides a model that has been degrading for two months.

Retraining, and how to decide

Three triggers, and only one of them is a schedule:

In practice: scheduled retraining as the floor, drift alerts as an early warning that triggers a human decision, performance-based as the authority when labels exist.

Whatever the trigger, the pipeline must be the same one that produced the current model, and the new model must clear the same gates:

def promote(candidate, champion, eval_set, holdout):
    checks = {
        "beats_champion": candidate.score(eval_set) > champion.score(eval_set),
        "no_segment_regression": all(
            candidate.score(s) >= champion.score(s) - 0.02
            for s in segments(eval_set)
        ),
        "recent_slice_ok": candidate.score(holdout.last_month) >= MIN_SCORE,
        "latency_ok": candidate.p99_latency_ms < LATENCY_BUDGET,
        "feature_versions_match": candidate.feature_versions == store.versions(),
    }
    if not all(checks.values()):
        raise PromotionBlocked({k: v for k, v in checks.items() if not v})
    return candidate

The no_segment_regression check is the one that saves you. An aggregate improvement that comes with a collapse on one region or customer tier is not an improvement — it is a trade you did not agree to make.

What to log for every prediction

You cannot debug what you did not record. The minimum:

log.info("prediction", extra={
    "request_id": request.id,
    "entity_id": request.user_id,
    "model_name": model.name,
    "model_version": model.version,
    "feature_versions": model.feature_versions,
    "features": features.to_dict(),         # the actual values used
    "feature_ages_s": feature_ages,
    "score": float(score),
    "threshold": THRESHOLD,
    "decision": decision,
    "latency_ms": elapsed_ms,
})

Logging the feature values that were actually used is what makes every later investigation possible: skew detection, drift attribution, and answering "why did the model do that on 14 March". Without it, every incident is archaeology.

Sample if volume demands it — but sample the logs, never the fields.

Frequently asked questions

What is training-serving skew?

When features are computed differently for training and for inference — different code, different filters, different timezones, different freshness. The model receives inputs from a distribution it was not trained on, and degrades silently. The fix is one feature implementation used by both paths.

How often should I retrain?

Start with a schedule matched to how fast your domain moves, then let drift and performance monitoring adjust it. Scheduled retraining alone either wastes effort or reacts too late; monitoring tells you which. Whatever the trigger, gate promotion on per-segment metrics, not only the aggregate.

What is a feature store actually for?

Two things: guaranteeing that training and serving use the same feature definitions, and supporting point-in-time correct historical lookups so training data reflects what was knowable at the time. If it is only a cache, it is solving the easy half.

How do I detect model drift without labels?

Monitor input feature distributions and the prediction distribution with PSI or a KS test against a reference window. Above about 0.2 PSI is a real shift. A sudden change in positive rate is the fastest label-free signal that something upstream broke.

Should I deploy a new model straight to production?

No. Shadow first — run it on real traffic with its output discarded — for at least one full weekly cycle, which catches crashes, latency regressions and feature incompatibility. Then canary a small share of traffic, routed by a stable user hash, and compare on the business outcome.

Next reading on Sythra Articles

• ML model evaluation mistakes
• Data leakage: the kinds that survive your split
• Agentic AI architecture