Python k-NN Exercises: Distance, Votes, and Scaling
k-NN is a good first classifier because every decision is visible. You can inspect the distance, the neighbors, the vote, and the evaluation. These exercises make each of those steps testable.
Write the distance once
Keep one distance function and reuse it for ranking and evaluation. A small function is easier to test than distance logic embedded in a loop. Check the dimensionality of the points before comparing them.
def distance(a, b):
return sum((x - y) ** 2 for x, y in zip(a, b)) ** 0.5
print(distance((0, 0), (3, 4)))A reusable measurement
Rank points with their labels
Sort (point, label) pairs by distance instead of sorting points and looking labels up later. This keeps the prediction attached to the evidence and avoids index mistakes after a sort.
def distance(a, b):
return sum((x - y) ** 2 for x, y in zip(a, b)) ** 0.5
points = [((1, 1), "a"), ((4, 4), "b"), ((1, 2), "a")]
target = (1, 1.5)
ranked = sorted(points, key=lambda item: distance(target, item[0]))
print(ranked[:2])Labels travel with points
Vote with a deterministic tie rule
Take the first k labels and count them. If two labels tie, use a rule such as the label of the nearest tied point. A random or hash-order tie makes the same input produce different predictions.
from collections import Counter
labels = ["a", "b", "a", "c"]
winner = Counter(labels).most_common(1)[0][0]
print(winner)A reproducible vote
Scale before measuring
Distance treats every unit as equal. If income is measured in thousands and age in years, income dominates. Standardize features using training statistics, then apply the same transform to validation and test data.
def distance(a, b):
return sum((x - y) ** 2 for x, y in zip(a, b)) ** 0.5
raw = distance((30, 30000), (30, 31000))
scaled = distance((0.3, 0.3), (0.3, 1.0))
print(round(raw, 1), round(scaled, 1))Units change the answer
Evaluate with held-out points
A training-set prediction is not an evaluation. Use a train/test split or leave-one-out for tiny datasets. Report the baseline too: predicting the majority class is often surprisingly strong.
data = [((0, 0), "a"), ((1, 0), "a"), ((5, 5), "b")]
correct = 0
for i, (point, label) in enumerate(data):
train = data[:i] + data[i + 1:]
nearest = min(train, key=lambda item: (item[0][0] - point[0]) ** 2 + (item[0][1] - point[1]) ** 2)
correct += nearest[1] == label
print(correct, len(data))Leave one out
Your practice checklist
- Use one tested distance function for ranking and evaluation.
- Keep labels attached to their points while sorting.
- Choose k and a deterministic tie rule before evaluating.
- Fit scaling on training data only.
- Compare against a majority-class baseline.
Common questions
How do I choose k?
Start with a small odd number for binary problems, then evaluate several values on validation data. Very small k is noisy; very large k can wash out local structure.
Why standardize rather than normalize?
Either can work. Standardization centers and scales by standard deviation; min-max normalization maps to a fixed range. The important part is fitting the transform on training data only.
Is k-NN expensive?
Prediction compares the new point with every stored training point, so large datasets need approximate indexes or a different model.
Practice k-NN
12 questions on distance, neighbors, voting, scaling, and held-out evaluation.
Keep reading: