The line was fitted by formula; now Maya wants to find it by walking downhill. The learning rate decides whether she arrives or overshoots. What happens when it is too large?
Gradient descent is a loop: measure the loss, compute the direction that increases it, then move the other way. The idea is small enough to implement with a few lines of Python, and the learning rate makes its failure modes visible. You will also add momentum and see why feature scale changes the gradient.
Start with a loss
A loss turns a model's mistakes into one number. For a line through the origin, mean squared error compares each prediction w * x with the real y. The model cannot improve until the loss defines what better means.
def loss(w, xs, ys):
return sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / len(xs)
print(loss(1, [1, 2], [2, 4]))One number for the current model
Compute the gradient
The derivative tells you the slope of the loss with respect to w. A negative gradient means increasing w lowers the loss, so the update subtracts the gradient. The factor 2 comes from differentiating the square.
def gradient(w, xs, ys):
return 2 * sum((w * x - y) * x for x, y in zip(xs, ys)) / len(xs)
print(gradient(1, [1, 2], [2, 4]))The direction of steepest increase
Take a step and repeat
The learning rate controls step size. Each iteration recomputes the gradient at the new w, then updates again. On this clean line the estimate moves toward 2 without ever needing to know the answer in advance.
def gradient(w, xs, ys):
return 2 * sum((w * x - y) * x for x, y in zip(xs, ys)) / len(xs)
xs = [1, 2]
ys = [2, 4]
w = 1.0
for _ in range(3):
w -= 0.1 * gradient(w, xs, ys)
print(round(w, 3))Small steps toward a better fit
The learning rate can break the loop
A learning rate that is too small wastes iterations. One that is too large can overshoot and diverge. Always inspect the loss curve, set a step budget, and stop when the update or improvement becomes negligible.
def gradient(w, xs, ys):
return 2 * sum((w * x - y) * x for x, y in zip(xs, ys)) / len(xs)
def train(learning_rate, xs, ys):
w = 1.0
for _ in range(5):
w -= learning_rate * gradient(w, xs, ys)
return w
xs = [1, 2]
ys = [2, 4]
print(round(train(0.1, xs, ys), 2))
print(abs(train(1.5, xs, ys)) > 1000)Same data, very different behavior
Momentum smooths repeated directions
Momentum keeps a velocity that accumulates past gradients. It can move faster through a consistent slope and damp noisy zigzags, but it can also overshoot near a minimum. Momentum is another hyperparameter to validate, not a guarantee of faster convergence.
def gradient(w, xs, ys):
return 2 * sum((w * x - y) * x for x, y in zip(xs, ys)) / len(xs)
xs = [1, 2]
ys = [2, 4]
w = 0.0
velocity = 0.0
for _ in range(10):
velocity = 0.9 * velocity - 0.1 * gradient(w, xs, ys)
w += velocity
print(round(w, 3))Accumulate direction before stepping
Feature scale changes the gradient
Multiplying an input by ten changes the loss surface and the gradient by a large factor. The same learning rate is no longer equivalent. Standardize features or tune the rate per scale, and check the first few loss values before a long run.
def gradient(w, xs, ys):
return 2 * sum((w * x - y) * x for x, y in zip(xs, ys)) / len(xs)
for factor in [1, 10]:
xs = [factor, 2 * factor]
ys = [2 * factor, 4 * factor]
print(factor, round(gradient(1.0, xs, ys), 1))The same rate is not scale-free
Key points
- A loss converts prediction errors into an optimization target.
- The gradient points toward increasing loss, so updates subtract it.
- The learning rate controls how far each update moves.
- Too large a rate can diverge; too small a rate can be slow.
- Track loss and use a step budget or convergence threshold.
- Momentum accumulates direction; feature scaling changes the effective learning rate.
Common mistakes
- Adding the gradient when the goal is to minimize the loss.
- Reusing the old gradient after changing the parameters.
- Judging training only by the final parameter instead of the loss curve.
- Reusing a learning rate after changing feature units and assuming convergence is unchanged.
Run ten updates with learning rate 0.1 on the same data and print the final weight rounded to three decimals.
Try it yourself
Change the values, predict the output, then run the code. The Python runtime is downloaded the first time you press Run, so the first run can take a few seconds.
Practice this lesson
12 questions with immediate explanations.