Model Monitoring Guide

A guide to monitoring ML models in production, detecting drift, measuring performance, and knowing when to retrain.

Table of Contents

1. Types of Drift
2. Statistical Tests for Drift Detection
3. Monitoring Metrics
4. Monitoring Tools
5. Code Examples with Evidently AI
6. When to Retrain
7. Interview Q&A
8. References

Types of Drift

Data Drift (Covariate Shift)

The distribution of input features changes while the relationship between inputs and outputs remains the same.

P_train(X) ≠ P_prod(X)
P_train(Y|X) = P_prod(Y|X)

Example: A loan approval model trained on pre-pandemic data sees different income and debt distributions post-pandemic.

Detection: Statistical tests on feature distributions (KS test, PSI).

Concept Drift

The relationship between inputs and outputs changes.

P_train(Y|X) ≠ P_prod(Y|X)

Example: Fraud patterns evolve, transactions that were previously legitimate are now fraudulent.

Detection: Requires ground truth labels (delayed feedback). Monitor proxy metrics or use windowed holdout sets.

Types of concept drift:

• Sudden drift: Abrupt change (e.g., system update, rule change)
• Gradual drift: Slow shift over time
• Recurring drift: Seasonal patterns (e.g., holiday fraud patterns)
• Incremental drift: Monotonic progression

Model Drift / Performance Drift

The downstream effect of data or concept drift: the model's predictions become less accurate or useful over time.

Detection: Monitor business KPIs, accuracy metrics (when labels available), and prediction score distributions.

Label Shift (Prior Probability Shift)

The distribution of the target variable changes while the class-conditional distribution stays the same.

P_train(Y) ≠ P_prod(Y)
P_train(X|Y) = P_prod(X|Y)

Example: The base rate of fraud increases during an economic crisis.

Statistical Tests for Drift Detection

Kolmogorov-Smirnov (KS) Test

Tests whether two continuous distributions are drawn from the same distribution.

Null hypothesis: The two distributions are identical.

from scipy import stats
import numpy as np

# Reference distribution (training data)
reference = np.random.normal(0, 1, 1000)

# Current distribution (production data)
current = np.random.normal(0.5, 1.2, 500)

ks_stat, p_value = stats.ks_2samp(reference, current)

print(f"KS Statistic: {ks_stat:.4f}")
print(f"P-value: {p_value:.4f}")

if p_value < 0.05:
    print("DRIFT DETECTED: distributions are significantly different")
else:
    print("No significant drift detected")

Interpretation:

• KS statistic: 0 = identical distributions, 1 = completely different
• p-value < 0.05: reject null hypothesis → drift detected

Population Stability Index (PSI)

Measures how much a distribution has shifted from a reference distribution. Widely used in banking/credit risk.

import numpy as np

def calculate_psi(reference, current, buckets=10):
    """Calculate PSI between reference and current distributions."""
    # Create bins based on reference distribution
    breakpoints = np.percentile(reference, np.linspace(0, 100, buckets + 1))
    breakpoints[0] = -np.inf
    breakpoints[-1] = np.inf

    ref_counts = np.histogram(reference, bins=breakpoints)[0]
    cur_counts = np.histogram(current, bins=breakpoints)[0]

    # Convert to proportions, avoid division by zero
    ref_pct = ref_counts / len(reference)
    cur_pct = cur_counts / len(current)

    # Avoid log(0)
    ref_pct = np.where(ref_pct == 0, 0.0001, ref_pct)
    cur_pct = np.where(cur_pct == 0, 0.0001, cur_pct)

    psi_values = (cur_pct - ref_pct) * np.log(cur_pct / ref_pct)
    psi = np.sum(psi_values)
    return psi

psi = calculate_psi(reference_data, production_data)

if psi < 0.1:
    status = "No significant change"
elif psi < 0.25:
    status = "Moderate change: investigate"
else:
    status = "Significant change: retrain required"

print(f"PSI: {psi:.4f}, {status}")

Chi-Squared Test (Categorical Features)

from scipy.stats import chi2_contingency
import numpy as np

# For categorical features
reference_counts = np.array([150, 200, 100, 50])
current_counts = np.array([120, 250, 80, 30])

# Build contingency table
contingency = np.array([reference_counts, current_counts])

chi2, p_value, dof, expected = chi2_contingency(contingency)

print(f"Chi-squared: {chi2:.4f}")
print(f"P-value: {p_value:.4f}")
print(f"Degrees of freedom: {dof}")

if p_value < 0.05:
    print("DRIFT DETECTED: categorical distribution has shifted")

Jensen-Shannon Divergence

from scipy.spatial.distance import jensenshannon
import numpy as np

def js_divergence(p, q, bins=50):
    """Jensen-Shannon divergence between two distributions."""
    hist_p, bin_edges = np.histogram(p, bins=bins, density=True)
    hist_q, _ = np.histogram(q, bins=bin_edges, density=True)

    # Normalize
    hist_p = hist_p / hist_p.sum() + 1e-10
    hist_q = hist_q / hist_q.sum() + 1e-10

    # scipy returns the JS *distance* (square root of the divergence);
    # base=2 bounds it to [0, 1], so squaring gives the divergence in [0, 1]
    return jensenshannon(hist_p, hist_q, base=2) ** 2

js = js_divergence(reference, current)
print(f"JS Divergence: {js:.4f}")  # 0 = identical, 1 = completely different (base 2)

Monitoring Metrics

Infrastructure Metrics

Data Quality Metrics

Model Performance Metrics

Monitoring Tools

Code Examples with Evidently AI

Installation

pip install evidently

The examples below use the current API (Evidently 0.7+). The pre-0.7 API (evidently.report, evidently.metric_preset, TestSuite, column_mapping) is only available under evidently.legacy and appears in many older tutorials.

Data Drift Report

import pandas as pd
from evidently import Report
from evidently.presets import DataDriftPreset, DataSummaryPreset

# Reference data (training distribution)
reference_data = pd.read_csv("train_data.csv")

# Current data (recent production data)
current_data = pd.read_csv("production_data.csv")

# Create a drift report
report = Report([
    DataDriftPreset(),
    DataSummaryPreset(),
])

# run() returns a Snapshot with the results; current data comes first
snapshot = report.run(current_data=current_data, reference_data=reference_data)

# Save HTML report
snapshot.save_html("drift_report.html")

# Get results as dict: each metric has "metric_name" and "value"
results = snapshot.dict()
drifted = results["metrics"][0]["value"]  # DriftedColumnsCount: {"count": ..., "share": ...}
print(f"Drifted columns: {drifted}")

Column-Level Drift

from evidently import Report
from evidently.metrics import ValueDrift

# Check specific columns
report = Report([
    ValueDrift(column="age"),
    ValueDrift(column="income"),
    ValueDrift(column="credit_score"),
])

snapshot = report.run(current_data=current_data, reference_data=reference_data)

for metric in snapshot.dict()["metrics"]:
    # value is the drift score of the auto-selected test (e.g. a p-value or a distance)
    print(f"{metric['metric_name']}: score={metric['value']:.4f}")

Model Performance Monitoring

from evidently import BinaryClassification, DataDefinition, Dataset, Report
from evidently.presets import ClassificationPreset

# When you have ground truth labels (delayed feedback).
# DataDefinition replaces the old column_mapping argument.
definition = DataDefinition(
    classification=[BinaryClassification(target="label", prediction_labels="prediction")]
)
current_ds = Dataset.from_pandas(current_with_labels, data_definition=definition)
reference_ds = Dataset.from_pandas(reference_with_labels, data_definition=definition)

report = Report([ClassificationPreset()])
snapshot = report.run(current_data=current_ds, reference_data=reference_ds)
snapshot.save_html("classification_report.html")

Continuous Monitoring Pipeline

from evidently import Report
from evidently.metrics import DriftedColumnsCount
from evidently.tests import lt

def run_monitoring_check(reference_data, current_data, threshold=0.3):
    """Run drift checks and return alert status."""

    # Tests are attached to metrics (TestSuite was folded into Report)
    report = Report([
        DriftedColumnsCount(
            tests=[lt(3)],                # Fail if 3 or more columns drift
            share_tests=[lt(threshold)],  # Fail if the drifted share reaches 30%
        ),
    ])

    snapshot = report.run(current_data=current_data, reference_data=reference_data)

    results = snapshot.dict()
    all_passed = all(
        test["status"] == "SUCCESS"
        for test in results["tests"]
    )

    return all_passed, results

passed, results = run_monitoring_check(reference_data, production_data)
if not passed:
    print("ALERT: Model monitoring check failed, investigate drift")
    # Send alert to Slack, PagerDuty, etc.

When to Retrain

Trigger Conditions

Retraining Strategies

Full retraining: Train from scratch on all available data. Safe but slow.

Incremental/online learning: Update model weights with new data only. Fast but requires algorithms that support it.

Windowed retraining: Train on a sliding window of recent data. Balances historical and recent patterns.

# Example: Triggering retraining based on PSI
def check_and_trigger_retrain(psi_scores, threshold=0.25):
    """Check PSI scores and trigger retraining if needed."""
    drifted_features = [
        feature for feature, psi in psi_scores.items()
        if psi >= threshold
    ]

    if drifted_features:
        print(f"Drift detected in: {drifted_features}")
        print("Triggering retraining pipeline...")
        # Trigger Airflow DAG, Kubeflow pipeline, etc.
        trigger_training_pipeline(
            reason="drift_detected",
            affected_features=drifted_features
        )
        return True
    return False

Interview Q&A

Q1: What is the difference between data drift and concept drift? 🟡 Intermediate

Data drift (covariate shift) is when the input feature distribution changes P(X) while the relationship P(Y|X) remains the same. Concept drift is when the relationship between inputs and outputs changes P(Y|X). Both lead to model degradation but require different interventions. Data drift can often be handled by retraining on new data; concept drift may require rethinking features or model architecture.

Q2: What statistical tests would you use for drift detection? 🟡 Intermediate

For continuous features: KS test (returns p-value and D-statistic) or PSI (interpretable thresholds). For categorical features: Chi-squared test. For probability distributions: Jensen-Shannon divergence or Wasserstein distance. The choice depends on feature type and interpretability requirements. PSI is preferred in banking because of its clear thresholds (< 0.1, 0.1-0.25, > 0.25).

Q3: How do you monitor a model when you don't have ground truth labels immediately? 🔴 Advanced

When labels are delayed (e.g., fraud confirmed days later, loan default confirmed months later):

1. Monitor input feature distributions using statistical tests (proxy for data drift)
2. Monitor prediction score distributions: shifts indicate potential concept drift
3. Monitor business proxy metrics (transaction decline rates, user complaints)
4. Use a held-out labeled set for periodic evaluation
5. Use model confidence scores as a proxy metric
6. Once labels arrive, run retrospective evaluation and trigger retraining

Q4: What is PSI and what are its thresholds? 🟡 Intermediate

PSI (Population Stability Index) measures distribution shift: PSI = Σ (Actual% - Expected%) × ln(Actual%/Expected%). Thresholds: PSI < 0.1 (no significant change), 0.1 ≤ PSI < 0.25 (moderate change, investigate), PSI ≥ 0.25 (significant change, retrain). PSI is widely used in credit risk because it is interpretable and standardized.

Q5: What monitoring would you set up for a fraud detection model? 🔴 Advanced

• Feature drift: Monitor all input features (transaction amount, merchant category, time of day) using PSI and KS tests
• Score distribution: Watch for shifts in the fraud probability score distribution
• Alert rate: Monitor % of transactions flagged as fraud, anomalies indicate drift
• Business metrics: Track false positive rate (customer friction) and false negative rate (missed fraud)
• Latency: Ensure real-time scoring meets SLA (< 50ms)
• Feedback loop: When fraud confirmed, update training data and check if model would have caught it

References

• Evidently AI Documentation
• WhyLogs Documentation
• Arize AI Blog on Model Monitoring
• Chip Huyen: ML Monitoring (book chapter)
• PSI and CSI: Understanding Population Stability Index