Python model monitoring and drift checks

A model that worked yesterday can fail quietly today when inputs or behavior change. Monitoring compares a live window with a trusted baseline, uses a threshold that limits noise, and records aggregates instead of personal data. These small checks catch drift before they become incidents. You will also compare binned distributions with PSI and separate input drift from outcome drift.

Compare live inputs with a baseline

Start with a metric that is easy to explain, such as the mean of a numeric feature. Store a baseline window from a period you trust, then compare a recent window. A difference is a signal to investigate, not proof that the model is broken.

baseline = [10, 12, 14]
live = [13, 15, 17]
shift = sum(live) / len(live) - sum(baseline) / len(baseline)
print(round(shift, 2))

A simple mean shift

Put the gap in baseline units

A raw difference is hard to compare across features with different scales. Divide by the baseline standard deviation to express the shift in familiar units. A larger absolute value is a stronger signal, but the threshold still depends on the domain.

import statistics

baseline = [10, 12, 14]
live = [13, 15, 17]
gap = (sum(live) / len(live) - sum(baseline) / len(baseline)) / statistics.pstdev(baseline)
print(round(gap, 2))

Three units is about 1.84 baseline deviations

Require enough evidence before alerting

A tiny live sample can produce a dramatic-looking average by chance. Require a minimum sample size, use a threshold, and handle a zero baseline deviation explicitly. An alert should lead to a documented check, not an automatic model change.

import statistics

def should_alert(baseline, live, min_n=5, threshold=1.0):
    if len(live) < min_n:
        return False
    baseline_mean = sum(baseline) / len(baseline)
    baseline_std = statistics.pstdev(baseline)
    if not baseline_std:
        return False
    gap = (sum(live) / len(live) - baseline_mean) / baseline_std
    return abs(gap) >= threshold

baseline = [10, 11, 12, 13, 14]
print(should_alert(baseline, [14, 15, 16, 17, 18]))
print(should_alert(baseline, [15, 16]))

Small samples do not trigger alerts

Monitor aggregates, not identities

Track rates and counts rather than raw user text, prompts, or identifiers. A shift in the positive-prediction rate can reveal changed inputs or model behavior without exposing personal data. Keep retention and access rules explicit.

baseline = [0, 0, 0, 0, 1]
live = [0, 0, 1, 1, 1]
base_rate = sum(baseline) / len(baseline)
live_rate = sum(live) / len(live)
print(round(live_rate - base_rate, 2))

A rate, not a person

Binned PSI compares distributions

The population stability index compares the share of values in fixed bins between a baseline and a live window. Small shares are replaced with a floor to avoid division by zero. PSI is a screening signal, not proof that the model is wrong; investigate the bins that moved.

import math

def shares(values, edges):
    counts = [0] * (len(edges) - 1)
    for value in values:
        for i in range(len(edges) - 1):
            if edges[i] <= value < edges[i + 1]:
                counts[i] += 1
                break
    total = sum(counts)
    return [count / total for count in counts]

edges = [0, 12, 16, 20, 30]
base = shares([10, 12, 14, 16, 18], edges)
live = shares([14, 16, 18, 20, 22], edges)
psi = sum((l - b) * math.log(max(l, 0.001) / max(b, 0.001)) for b, l in zip(base, live))
print([round(x, 2) for x in base])
print([round(x, 2) for x in live])
print(round(psi, 3))

Compare shares, not identities

Separate input drift from outcome drift

Input drift means the feature distribution moved. Outcome drift means the model's accuracy or error cost changed after labels arrive. They can occur together or separately, so monitor both and keep the alert threshold tied to a minimum sample size.

def accuracy(actual, predicted):
    return sum(a == p for a, p in zip(actual, predicted)) / len(actual)

baseline_actual = [1, 0, 1, 1, 0, 1]
baseline_pred = [1, 0, 1, 1, 0, 1]
live_actual = [1, 1, 0, 1, 0, 1]
live_pred = [1, 1, 0, 1, 0, 0]
print("baseline", round(accuracy(baseline_actual, baseline_pred), 2))
print("live", round(accuracy(live_actual, live_pred), 2))

Inputs and labels are different signals

Try it yourself

Change the values, predict the output, then run the code. The Python runtime is downloaded the first time you press Run, so the first run can take a few seconds.