Python model monitoring and drift checks Exercises With Answers

What is a monitoring baseline?

1. A list of user identifiers
2. A trusted reference window for comparisoncorrect
3. A second model with more parameters
4. The final production prediction

Answer: A trusted reference window for comparison

Why: The baseline records what normal looked like during a period you trust.

Why require a minimum live sample size before alerting?

1. It increases the learning rate
2. It makes the model more accurate
3. It removes the baseline
4. Small samples can look extreme by chancecorrect

Answer: Small samples can look extreme by chance

Why: A few unusual observations can move an average a lot without indicating real drift.

What is the difference between input drift and concept drift?

1. Input drift only affects CSS
2. Concept drift only affects the learning rate
3. Input drift changes the data; concept drift changes the input-target relationshipcorrect
4. They are exactly the same

Answer: Input drift changes the data; concept drift changes the input-target relationship

Why: Inputs can shift while the relationship stays stable, or the relationship itself can change.

What is the output?

baseline = [10, 12, 14]
live = [13, 15, 17]
print(sum(live) / len(live) - sum(baseline) / len(baseline))

1. 12.0
2. 15.0
3. 3.0correct
4. 1.0

Answer: 3.0

Why: The live mean is 15.0 and the baseline mean is 12.0, so the shift is 3.0.

Complete the baseline standard deviation call.

import statistics

baseline = [10, 12, 14]
print(round(statistics.____(baseline), 2))

Answer: pstdev

Why: statistics.pstdev computes the population standard deviation.

Complete the alert comparison.

gap = 1.84
threshold = 1.0
print(abs(gap) >= ____)

Answer: threshold

Why: The absolute gap is compared with the chosen threshold.

Complete the minimum-sample guard.

def enough(live, min_n=5):
    if len(live) < ____:
        return False
    return True

print(enough([1, 2, 3]))

Answer: min_n

Why: The function returns early when the live window has fewer than the required observations.

Complete the rate denominator.

live = [0, 0, 1, 1, 1]
print(sum(live) / ____(live))

Answer: len

Why: A rate divides the count of positive outcomes by the number of observations.

What does this code print?

baseline = [10, 12, 14]
live = [13, 15, 17]
print(round(sum(live) / len(live) - sum(baseline) / len(baseline), 2))

Answer: 3.0

Why: The live mean is 15.0 and the baseline mean is 12.0, so the rounded shift is 3.0.

What does the program output?

import statistics
baseline = [10, 12, 14]
live = [13, 15, 17]
gap = (sum(live) / len(live) - sum(baseline) / len(baseline)) / statistics.pstdev(baseline)
print(round(gap, 2))

Answer: 1.84

Why: The mean shift of 3.0 divided by the baseline population standard deviation gives 1.84.

What appears on the screen?

import statistics

def should_alert(baseline, live, min_n=5, threshold=1.0):
    if len(live) < min_n:
        return False
    baseline_std = statistics.pstdev(baseline)
    if not baseline_std:
        return False
    gap = (sum(live) / len(live) - sum(baseline) / len(baseline)) / baseline_std
    return abs(gap) >= threshold

print(should_alert([10, 11, 12, 13, 14], [14, 15, 16, 17, 18]))

Answer: True

Why: The live window has five observations and the standardized shift exceeds 1.0, so the result is True.

What is printed?

baseline = [0, 0, 1, 1]
live = [0, 1, 1, 1]
print(round(sum(live) / len(live) - sum(baseline) / len(baseline), 2))

Answer: 0.25

Why: The live positive rate is 0.75 and the baseline rate is 0.5, so the difference is 0.25.