Python Model Monitoring Exercises: Drift and Alerts
Monitoring is a set of small comparisons, not a dashboard full of numbers. These exercises build a baseline, measure a shift, decide whether there is enough evidence to alert, and record only the aggregates you need.
Exercise: compare a live mean with a baseline
Choose one numeric feature and one baseline window. The difference in means is a simple first signal. It is easy to explain, but it has no sense of how variable the baseline was.
baseline = [10, 12, 14]
live = [13, 15, 17]
shift = sum(live) / len(live) - sum(baseline) / len(baseline)
print(round(shift, 2))A three-unit shift
Exercise: standardize the gap
Divide the mean shift by the baseline standard deviation. The result is easier to compare across features, but it still needs a domain-specific threshold.
import statistics
baseline = [10, 12, 14]
live = [13, 15, 17]
gap = (sum(live) / len(live) - sum(baseline) / len(baseline)) / statistics.pstdev(baseline)
print(round(gap, 2))About 1.84 baseline deviations
Exercise: require enough evidence
A two-item live window can produce an alarming mean by chance. Gate the alert on a minimum sample size, then apply the threshold. Return a reason so the alert can be debugged.
import statistics
def check(baseline, live, min_n=5, threshold=1.0):
if len(live) < min_n:
return "collect more data"
baseline_std = statistics.pstdev(baseline)
if not baseline_std:
return "baseline has no variation"
gap = (sum(live) / len(live) - sum(baseline) / len(baseline)) / baseline_std
return "alert" if abs(gap) >= threshold else "watch"
baseline = [10, 11, 12, 13, 14]
print(check(baseline, [14, 15, 16, 17, 18]))
print(check(baseline, [15, 16]))Alert with a reason
Exercise: track a prediction rate
Rates are often more useful than raw counts because traffic changes. Compare the share of positive predictions, errors, or fallbacks between windows. A rate still needs a sample-size guard.
baseline = [0, 0, 0, 0, 1]
live = [0, 0, 1, 1, 1]
base_rate = sum(baseline) / len(baseline)
live_rate = sum(live) / len(live)
print(round(live_rate - base_rate, 2))A rate difference
Exercise: log aggregates, not identities
A count table is enough to monitor many failures. Do not log raw prompts, user text, or identifiers unless the need is explicit and the retention policy is approved. Aggregate first.
events = ["ok", "ok", "error", "ok", "error"]
counts = {}
for event in events:
counts[event] = counts.get(event, 0) + 1
total = len(events)
print({key: round(value / total, 2) for key, value in counts.items()})Counts and rates, not raw records
Your practice checklist
- Record a baseline window and the date it represents.
- Monitor both inputs and outputs, not only accuracy.
- Standardize shifts before comparing different features.
- Require a minimum sample size before alerting.
- Log aggregates and set a retention policy.
Common questions
How often should I monitor?
Frequently enough to catch meaningful changes before users are harmed, but not so often that noise dominates. The right cadence depends on traffic and risk.
Does a drift alert mean the model is broken?
No. Drift is a signal to investigate. The model may still be accurate; the input distribution may simply have changed.
Can I log prompts to debug drift?
Only with a clear need, user consent where required, strict access controls, and a retention limit. Prefer aggregate statistics or redacted samples.
Practice monitoring signals
12 questions on baselines, standardized shifts, sample-size guards, alert thresholds, and aggregate metrics.
Keep reading: