Python Softmax Exercises: Stability, Temperature, Log Loss

Normalize the exponentials

Apply exp to each logit and divide by the sum. The output is positive and sums to one. Check the sum in tests, because a silent normalization mistake can make later metrics meaningless.

import math

def softmax(logits):
    exps = [math.exp(x) for x in logits]
    total = sum(exps)
    return [x / total for x in exps]

probs = softmax([1, 1])
print([round(p, 3) for p in probs])
print(round(sum(probs), 3))

A distribution, checked

Shift by the maximum

Large logits overflow before normalization. Subtracting the maximum shifts all values without changing their ratios. The result is mathematically the same and numerically safe.

import math

def stable_softmax(logits):
    largest = max(logits)
    exps = [math.exp(x - largest) for x in logits]
    total = sum(exps)
    return [x / total for x in exps]

print([round(p, 3) for p in stable_softmax([1000, 1000])])

Same answer, no overflow

Vary the temperature

Temperature divides the logits before softmax. Values below one sharpen the largest score; values above one flatten the distribution. Temperature does not add evidence; it changes how strongly the model commits.

import math

def softmax(logits, temperature=1.0):
    scaled = [x / temperature for x in logits]
    largest = max(scaled)
    exps = [math.exp(x - largest) for x in scaled]
    total = sum(exps)
    return [x / total for x in exps]

print([round(p, 3) for p in softmax([2, 0], 0.5)])
print([round(p, 3) for p in softmax([2, 0], 2.0)])

Sharper or flatter

Compute log loss for the true class

If the true class has predicted probability p, its log loss is -log(p). A confident correct prediction has low loss; a confident wrong prediction has a large loss. Clip probabilities away from zero before taking the logarithm.

import math

p_true = 0.8
loss = -math.log(max(p_true, 1e-12))
print(round(loss, 3))

A penalty for confidence

Argmax is a decision, not a truth claim

The highest probability gives a class label, but the full distribution may be close or uncertain. Report the top probability and the margin between the top two classes. For consequential decisions, add review or abstain when confidence is low.

probs = [0.46, 0.44, 0.10]
order = sorted(range(len(probs)), key=probs.__getitem__, reverse=True)
print(order[0])
print(round(probs[order[0]] - probs[order[1]], 2))

Winner plus margin

Your practice checklist

• Check that softmax outputs are positive and sum to one.
• Subtract the maximum logit before exponentiating.
• Use a positive temperature and document its effect.
• Clip probabilities before computing log loss.
• Report the top probability and margin, not only the argmax.

Common questions

Keep reading:

• Python softmax and temperature
• Python Exercises With Answers: How to Practice Without Copying
• Python Practice for Beginners: A Simple 4-Week Plan