Training the model is the easy part.

Every ML system that fails in production fails in one of a small number of well-known ways — and almost none of them are about the model.

Research code answers “does this idea work?” on a static dataset, once. A production ML system has to keep answering that question correctly as data drifts, upstream schemas change, and the same features have to be computed identically at training time and at inference time, months apart, by different code paths. The gap between those two problems is where most of the actual engineering lives.

Feature stores and point-in-time correctness

The most common way a research result silently fails to reproduce in production is a mismatch between how a feature was computed during training and how it is computed at serving time — training-serving skew. A feature store centralizes feature computation into a single definition consumed by both paths, eliminating the duplicate-implementation risk entirely.

The harder problem it solves is point-in-time correctness: when building a training set, every feature value must reflect only information that would have actually been available at that historical timestamp. This is the same discipline as the look-ahead-bias pitfall in the time series essay, generalized into infrastructure — a feature store with a correct point-in-time join makes the bug structurally hard to introduce, rather than relying on every researcher to avoid it by hand every time.

Model serving and registries

A model registry versions trained models alongside the exact code, data snapshot, and hyperparameters that produced them — reproducibility as a first-class artifact, not an afterthought reconstructed from git history and memory. Serving infrastructure on top of it typically supports:

• Shadow deployment. The new model runs alongside the current one on live traffic without its predictions taking effect, comparing outputs before any decision depends on it.
• Canary rollout. The new model takes a small, increasing share of traffic, with automatic rollback if key metrics degrade.
• Champion/challenger. Multiple models run concurrently in production, with routing and comparison built into the serving layer rather than bolted on after the fact.

Drift detection

Models degrade in production for two structurally different reasons, and conflating them leads to the wrong fix:

• Data (covariate) drift. The distribution of input features shifts, but the relationship between features and target is unchanged. Detected by comparing feature distributions over time — the Population Stability Index (PSI) and Kolmogorov-Smirnov tests are standard.
• Concept drift. The relationship between features and target itself changes — the same input now implies a different outcome. This is a regime change in the sense of the time series essay, and no amount of feature-distribution monitoring will catch it — it requires tracking model performance against ground truth directly, with whatever label latency that implies.

Experimentation: A/B testing as applied causal inference

A/B testing is the production system's version of the randomized controlled trial from the causal inference essay — randomization is what licenses the causal claim “this model change caused this metric change,” as opposed to the correlational claim that the metric merely moved after the change shipped. The engineering problem is making randomization, power analysis, and guardrail metrics (regressions you must catch even if the primary metric improves) a reusable platform capability rather than a one-off script per experiment.

Reproducibility as infrastructure

“It worked on my machine last quarter” is a production incident waiting to happen. Reproducibility requires versioning all three of the model's inputs together, not just the code:

1. Code — standard version control, but including the training pipeline, not just the model definition.
2. Data — snapshotted or content-addressed so a training run can be replayed against the exact data it saw.
3. Environment — pinned dependencies and, ideally, containerized execution, since numerical library versions can change results in ways that are silent and hard to trace.

Common pitfalls

• Training-serving skew. Feature logic implemented twice — once in the training pipeline, once in the serving path — that quietly diverges as one gets updated without the other.
• Silent pipeline breakage. An upstream schema change that doesn't crash the pipeline but changes a feature's meaning or units, degrading model performance without an obvious error to trace.
• Alert fatigue. Drift monitors tuned so sensitively that every alert gets ignored, which is operationally equivalent to having no monitoring at all.
• Metric gaming via retraining cadence. Retraining frequently enough to always look good on the most recent window can mask a model that has stopped generalizing at all.

The path forward

A production ML system that survives contact with reality, in order:

1. Centralize feature computation in a store with enforced point-in-time correctness, shared by training and serving.
2. Version code, data, and environment together for every trained model; deploy via shadow or canary before full rollout.
3. Monitor data drift and concept drift separately, and gate retraining behind human review, not an automatic trigger.
4. Treat A/B testing as causal-inference infrastructure — build it once, reuse it for every launch decision.