Why MLOps? ML technical debt and maturity levels
Moving a Jupyter notebook to production is one of the most underestimated challenges in machine learning. Models degrade over time (drift), dependencies evolve, data changes, and without rigorous practices, maintaining a model in production quickly turns into a technical-debt sinkhole.
MLOps (Machine Learning Operations) addresses this by applying DevOps principles — automation, reproducibility, monitoring, CI/CD — to the ML lifecycle. The goal: deploy models quickly and reliably, and keep them performing over time without constant manual intervention.
MLOps maturity levels: 0 to 2
Google Cloud formalized three MLOps maturity levels. Level 0 — Manual: Data Scientists work in Jupyter notebooks, deployment is an ad hoc process (uploading a .pkl file, a manual script), there is no automated training pipeline, no monitoring, and production models are not traceable. This is where most nascent ML projects sit.
Level 1 — Pipeline automation: training is automated in a reproducible pipeline (Kubeflow, Vertex AI Pipelines, Prefect), and Continuous Training (CT) automatically retrains the model on new data based on a schedule or a drift trigger. The deployed model is traceable (model registry). Level 2 — Full CI/CD: on top of CT, the training pipeline itself is versioned and tested in CI. Any change to the feature engineering code or the model triggers a complete validation pipeline before promoting a new version to production. This is the target state for mission-critical ML systems.
The seminal paper 'Hidden Technical Debt in Machine Learning Systems' (Sculley et al., Google, NeurIPS 2015) showed that ML code itself represents only a small fraction of the system: configuration, data collection, feature verification, serving infrastructure, monitoring — all of this makes up the bulk of the real complexity. The authors identified ML-specific antipatterns: feature entanglement, hidden feedback loops, and correction of the real world (the model influences the very data it will later predict on).
Sculley et al. - Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015The lifecycle of an ML model
The ML lifecycle comprises several sequential but iterative phases, each of which can be automated and versioned. The key point is that these phases are not linear: evaluation results can call feature engineering choices back into question, and production monitoring can trigger a new training cycle.
1. Data preparation and validation
Collecting and validating source data (schema, distribution, completeness), cleaning and handling missing values, creating features (transformation, encoding, normalization), a train/validation/test split with attention to data leakage, and dataset versioning (DVC, Delta Lake, Iceberg). Data validation is often neglected yet critical: TFX Data Validation, Great Expectations or Soda let you define assertions on training data that automatically fail the pipeline if the data is abnormal (a missing column, an aberrant distribution, an excessive null rate).
Data leakage is the major trap of this phase: features built with information from the future (that would not be available at inference time) produce models that look artificially strong during evaluation but are unusable in production. A feature store with point-in-time correctness (see the next section) is the systematic solution.
2. Experimentation and model selection
Tracked experimentation: MLflow Tracking, Weights & Biases, Comet ML or Neptune let you log hyperparameters, metrics, artifacts and code for every run. Without tracking, comparing experiments becomes impossible. Hyperparameter tuning: Optuna (Bayesian), Ray Tune (distributed), Keras Tuner. Selection based on business metrics (not just loss) — a model with 99% precision on an imbalanced dataset can have zero recall on the minority class.
Experiments must be reproducible: a fixed environment (Docker or Conda with pinned versions), a fixed random seed, a versioned dataset. Reproducibility is the basic condition for comparing two experiments meaningfully and for debugging unexpected behavior in production.
3. Rigorous evaluation before deployment
A model does not get deployed just because it clears a precision threshold. Pre-deployment evaluation includes: performance tests by segment (algorithmic fairness — does the model behave differently across demographic groups?), evaluation on recent data outside the training period (temporal holdout), and robustness tests (perturbations, adversarial inputs, out-of-domain values).
Comparison against the model currently in production (champion/challenger) is essential: if the new model does not beat the existing one on business metrics, it does not ship. This automatic gate in the CI/CD pipeline prevents silent regressions.
4. Deployment and serving strategies
Deployment strategies: blue/green (instant switch, immediate rollback on regression, zero downtime), canary (progressive rollout at 5% → 20% → 100%, lets you catch regressions on limited real traffic), shadow mode (the new model makes predictions in parallel with no impact on users — ideal for comparing outputs before any production release), A/B testing (two versions serve different user segments to compare business performance under real conditions).
Serving formats: REST API (FastAPI + Docker — flexible, versatile), batch scoring (Spark, dbt + SQL — for cases where latency isn't critical), edge deployment (ONNX, TFLite, Core ML — models embedded on-device). The choice depends on the latency SLA (real time < 100ms vs. overnight batch), request volume and cost constraints.
CI/CD for ML: pipelines, model tests and model registry
ML CI/CD differs from classic application CI/CD: beyond code, you also need to version and test data, features and models. A complete ML CI/CD pipeline adds Continuous Training (CT) on top of classic CI/CD.
A typical ML CI/CD pipeline
Trigger (a push to the repo, a schedule, or a newly detected data batch) → data validation (schema, distribution, completeness via Great Expectations or TFX DV) → reproducible feature engineering → model training with a fixed seed → automatic evaluation (metrics vs. baseline and vs. the champion model in prod) → registration in the model registry if the gates pass → serving integration tests (p95 latency, output format, handling of edge-case inputs) → deployment to staging → automatic smoke tests → promotion to production per the chosen strategy (canary or blue/green).
Pipeline tools: GitHub Actions or GitLab CI for CI triggers and orchestration, Kubeflow Pipelines or Vertex AI Pipelines for distributed ML pipelines on Kubernetes, Prefect or Airflow for data workflows, ZenML (a stack-agnostic multi-tool ML framework). The recent trend is toward declarative, code-versionable pipelines (DAG as code).
ML testing: beyond metrics
ML testing spans several levels. Unit tests for features: each transformation function is tested with known inputs and expected outputs (edge values, nulls, unexpected types). Preprocessing pipeline tests: does the full pipeline produce the correct output schema on a small reference dataset? Behavioral tests of the model: does the model satisfy expected invariances? (e.g., changing race in a text feature should not change a credit-scoring model's prediction). Performance tests: does the model clear the minimum threshold on the holdout set, and does it beat the champion model?
Behavioral (or metamorphic) tests are particularly important for NLP models: minor input perturbations (synonym replacement, case changes) should not produce radical prediction changes. The Checklist library (Ribeiro et al., Microsoft, ACL 2020) formalizes this approach.
Model registry: versioning and model governance
A model registry is the centralized catalog of ML models: it stores artifacts (weights, preprocessing pipeline, input/output signature), metadata (evaluation metrics, dataset used, hyperparameters, the code's git commit), lifecycle stages (None → Staging → Production → Archived) and the history of transitions, with optional human approval. Without a registry, the 'model in prod' is often a .pkl file sitting somewhere on a server, with no traceability and no way to roll back.
MLflow Model Registry (open-source, integrates with Databricks), Vertex AI Model Registry (GCP), SageMaker Model Registry (AWS), Weights & Biases Registry and Comet ML Model Registry are the most widespread solutions. The choice often follows the cloud provider or the primary MLOps platform. The essential point: every production deployment must point to a version registered in the registry, with a complete audit trail.
ML CI/CD adds Continuous Training (CT): beyond testing and deploying code, models are automatically retrained on new data when a condition is met (drift detected, performance below a threshold, a new weekly batch). This is the loop that keeps models current without manual intervention. CT is distinct from classic CI/CD because it involves significant compute resources (GPU, large data volumes) — it does not trigger on every commit but on business or quality triggers.
Feature store: centralizing and reusing ML features
A feature store is a platform that centralizes the creation, storage, documentation and serving of ML features. It solves a key problem: preventing every team from recomputing the same features differently, with the risk of training/serving skew (features computed differently at training time and at inference time).
Offline/online architecture of a feature store
A feature store has two parts: the offline store (a batch database for training — S3 + Parquet, BigQuery, Snowflake, Delta Lake) and the online store (a low-latency database for real-time inference — Redis, DynamoDB, Bigtable, Cassandra). The feature pipeline synchronizes both: it computes features in batch, materializes them in the offline store, then propagates them to the online store for real-time serving.
Major tools: Feast (open-source, cloud-agnostic, integrates with Redis, BigQuery, Snowflake), Tecton (managed, enterprise, AWS and GCP), Vertex AI Feature Store (GCP-native, managed), SageMaker Feature Store (AWS-native), Databricks Feature Engineering (native integration with Delta Lake and Unity Catalog). The choice depends on your existing cloud ecosystem and budget.
Point-in-time correctness: avoiding data leakage
Point-in-time correctness is the guarantee that, when building a training dataset, every example uses only the features available at the time of the event (not features computed afterward). Without this guarantee, you create data leakage: if you join a user's profile features using their current value instead of their value at the time of purchase, you 'contaminate' the training examples with information from the future.
Feature stores implement point-in-time correctness through temporal queries: 'give me the feature values for this user, as they existed at this exact date and time.' Without a feature store, reproducing this guarantee manually with temporal SQL joins is error-prone and hard to audit. This is one of the main reasons organizations with several ML models in production invest in a feature store.
Training/serving skew happens when the features used during training are computed differently during inference (normalization with different statistics, different encoding, different time windows). A feature store guarantees that the same logic is applied in both contexts. Without a feature store, this kind of bug is silent and can degrade production performance with no obvious alert — the model responds, it doesn't crash, but its predictions become steadily less reliable.
ML monitoring: data drift, concept drift and alerting
ML monitoring is fundamentally different from application monitoring. An API can be 'up' (good latency, no 5xx errors) while producing incorrect predictions because of a change in input data. ML monitoring watches prediction quality, not just infrastructure.
The three types of drift and how to detect them
Data drift (covariate drift): the distribution of inputs changes relative to the training distribution. Example: a credit-scoring model was trained on pre-COVID data; income distributions and payment behaviors have since shifted. Detection: statistical tests (Kolmogorov-Smirnov for continuous variables, Chi-square for categorical ones, Population Stability Index — PSI — for credit scorecards).
Concept drift: the relationship between inputs and output changes — the world evolves, but the inputs stay the same. Example: 'using the internet in the evening' correlated with 'young' in 2005, but no longer does in 2025. Harder to detect because it requires production labels (ground truth). Performance drift: business metrics (precision, recall, RMSE, business KPIs like the incremental revenue generated by a recommendation model) degrade. Requires a production labeling pipeline and enough time to accumulate sufficient labels.
ML observability and monitoring tools
Evidently AI (open-source, HTML reports and Grafana dashboards, monitors feature distribution and performance metrics), Arize AI (managed platform, built-in SHAP explanations, segment analysis), WhyLabs (continuous monitoring with data profiles), Fiddler (enterprise, monitoring and explainability). Cloud platforms have their own solutions: Vertex AI Model Monitoring (GCP), SageMaker Model Monitor (AWS, detects data drift and model quality drift), Azure ML Data Drift. These tools generally integrate with the existing observability stack (Prometheus, Grafana, Datadog, PagerDuty for alerting).
ML observability goes beyond drift: it includes logging predictions and inputs in production (for debugging and retraining), monitoring serving latencies (p50, p95, p99), fairness metrics by segment, and feedback loops (real labels once they arrive). Keeping a log of predictions with their inputs lets you reconstruct a retraining dataset directly from real production data.
Two strategies. Schedule-based: automatic retraining every week or every month — simple to implement, but may retrain unnecessarily (GPU cost) or not fast enough (if drift moves quickly). Trigger-based: retraining fires when PSI exceeds a threshold (e.g., PSI > 0.2 on a key feature) or when performance drops below a threshold — more precise but requires a robust monitoring pipeline. In practice, mature teams combine both: a minimum schedule plus triggers for sudden drift.
When to adopt MLOps and when to avoid the complexity
MLOps is not right for every context. The goal is to match your MLOps maturity level to your actual needs — not too little (technical debt, models silently degrading), and not too much (unnecessary complexity that slows down iteration).
Criteria for moving up in MLOps maturity
Moving from level 0 to level 1 is justified when: the production model needs regular retraining (changing data, evolving users), several Data Scientists work on the same project and need to compare their experiments systematically, time-to-production repeatedly stretches past several weeks, or 'model crashed in prod' incidents have happened and can't be debugged for lack of traceability.
Moving to level 2 (full CI/CD) is justified for mission-critical ML systems: product recommendations, credit scoring, fraud detection, dynamic pricing — any model whose degradation has a direct, measurable impact on revenue or risk. For an internal analytics reporting model that runs once a week, level 1 is more than enough.
Choosing your MLOps stack based on context
Startup / small team: start with MLflow (open-source tracking + model registry) plus GitHub Actions for automation plus Evidently AI for monitoring. A minimalist stack, quick to set up, extensible. Scale-up on GCP: Vertex AI Pipelines + Vertex AI Model Registry + Vertex AI Feature Store + Cloud Monitoring — an integrated, managed, ops-free stack. Scale-up on AWS: SageMaker Pipelines + SageMaker Model Registry + SageMaker Feature Store + CloudWatch. Multi-cloud or on-premise enterprise: Kubeflow Pipelines (on Kubernetes) + MLflow + Feast + Seldon or BentoML for serving — an open-source, portable stack that requires more ops work.
The most popular platform in 2025 is Databricks (which natively integrates MLflow, Unity Catalog, Feature Engineering, and builds on Delta Lake) for organizations already on Spark or Delta. The primary selection criterion isn't the technology — it's adoption friction: the best tool is the one your team will actually use.
Just like Data Mesh theater, there is an 'MLOps theater': organizations that deploy Kubeflow, MLflow, a feature store and Evidently without actually having an ML model in production (or just one, rarely retrained). MLOps infrastructure is an investment justified by the volume of models in production and how often they're retrained. For a first model, a notebook plus a monthly retraining cron job can be a better fit than Kubeflow.
Anchoring MLOps with spaced repetition
MLOps combines infrastructure concepts (feature store, model registry, pipelines), methodology (ML CI/CD, Continuous Training, ML testing) and statistics (drift, statistical tests, PSI). The sheer number of tools and their interdependence makes passive memorization inefficient.
Memia's 'MLOps and ML Production' and 'Monitoring and Model Drift' flashcard decks cover the key distinctions for ML Engineer, senior Data Scientist and Head of Data Science interviews: MLOps maturity levels, the three types of drift, the feature store's role, the structure of a model registry, and deployment strategies.
The most tested themes: (1) The 3 types of drift and how to detect them. (2) Training/serving skew: what it is, how to avoid it. (3) The difference between blue/green, canary and shadow deployment. (4) The model registry's role: what it stores, why it's critical. (5) Point-in-time correctness: why it's necessary in a feature store. (6) MLOps maturity levels 0/1/2: characteristics and transition criteria.
Explore the Data & AI cluster
Frequently asked questions about MLOps
What is MLOps?
MLOps (Machine Learning Operations) is a set of practices combining DevOps, Data Engineering and Machine Learning to industrialize the ML model lifecycle. It covers versioning data and models, automated training and deployment pipelines, production monitoring, and retraining. The goal is to move from a notebook to a reliable, maintainable ML system over the long term.
What are the MLOps maturity levels?
Google Cloud defines 3 levels. Level 0 (manual): notebooks, ad hoc deployment, no monitoring. Level 1 (pipeline automation): automated training in a reproducible pipeline, Continuous Training on new data, model registry. Level 2 (full CI/CD): the training pipeline itself is versioned and tested in CI — any change to the feature engineering code or the model triggers a validation pipeline before promotion to production.
What is the difference between MLOps and DevOps?
DevOps automates the lifecycle of application code (build, test, deploy, monitor). MLOps extends these principles to ML: you must version data and models (not just code), test models (not just the application), monitor prediction quality (not just infrastructure), and manage retraining as data evolves. Continuous Training (CT) is the key ML-specific extension.
What is a feature store?
A feature store centralizes the creation, storage and serving of ML features. It includes an offline store for training (batch, high capacity: S3, BigQuery, Delta Lake) and an online store for real-time inference (low latency: Redis, DynamoDB). It guarantees point-in-time correctness (avoiding data leakage) and that features are computed the same way at training time and in production. Tools: Feast, Tecton, Vertex AI Feature Store, SageMaker Feature Store, Databricks Feature Engineering.
What is drift in ML?
Drift refers to a model's performance degrading due to changes. Data drift: the distribution of inputs changes vs. the training distribution (detected via KS test, PSI). Concept drift: the relationship between inputs and output changes — the world evolves (harder to detect, requires production labels). Performance drift: business metrics degrade (requires a labeling pipeline). Each type requires a tailored response.
What is training/serving skew?
Training/serving skew occurs when the features computed during training differ from those computed during inference (different normalization statistics, different encoding, different time windows). The model responds without error but its predictions silently degrade. A feature store using the same computation logic for training and serving eliminates this risk. It's one of the hardest bugs to catch without proper tooling.
What is a model registry?
A model registry centralizes ML models: artifacts (weights, preprocessing pipeline), metadata (metrics, dataset used, the code's git commit), stages (Staging → Production → Archived) and the history of transitions. Without a registry, production models aren't traceable and rollbacks are impossible. Tools: MLflow Model Registry, Vertex AI Model Registry, SageMaker Model Registry, Weights & Biases Registry.
What are the ML deployment strategies?
Blue/green: instant switch, immediate rollback, zero downtime. Canary: progressive rollout (5% → 100%) — catches regressions on limited real traffic. Shadow mode: parallel predictions with no user impact — ideal before any production release to compare outputs. A/B testing: two versions for different segments — compares business metrics under real conditions. The choice depends on business risk and volume.
What is point-in-time correctness?
Point-in-time correctness guarantees that, when building a training dataset, every example uses only the features available at the time of the event — not values computed after the fact. Without this guarantee, you create data leakage: the model looks strong during evaluation but fails in production. Feature stores implement this guarantee through temporal queries ('features for user X as they existed on March 15 at 2:35pm').
When should you not adopt MLOps?
Don't over-engineer for a first model or a team of 1-2 people. A notebook plus a monthly retraining cron job may be enough. The MLOps investment is justified when: several models are in production simultaneously, retraining needs to be frequent (fast-changing data), model degradation has a direct impact on revenue or risk, or several Data Scientists work in parallel and need to compare their experiments. Start with MLflow for tracking — that's the useful minimum.
Previous article: Data Mesh, Data Products and Data Contracts