Gradient Descent: Walking Downhill Blindfolded
In the last post, we built a way to measure exactly how wrong a guess is, using the cost function. Given any w and b, we can now compute a single number that tells us how badly that line fits our data.
But there's a problem we quietly ignored. Knowing that a guess is wrong doesn't tell you which way to fix it. If your cost is 8.5, should you increase w or decrease it? Should b go up or down? The cost function tells you how bad things are. It says nothing about what to do next.
This post is about solving exactly that problem.
Imagine You're Standing on a Hill, Blindfolded
Picture this. You're standing somewhere on a hilly landscape, and your goal is to reach the lowest point in the valley. There's just one catch. You're blindfolded. You can't see the landscape at all.
What would you actually do? Most people would do something like this. Feel the ground under your feet. Figure out which direction feels like it's sloping downward. Take a small step in that direction. Then stop, feel the ground again, and repeat.
You wouldn't take one giant leap and hope for the best. A blindfolded giant leap could easily send you flying past the valley and up the opposite hill. Small, careful steps, checked again and again, is how you'd actually get down safely.
This is almost exactly what gradient descent does. Except instead of a hill made of dirt and grass, the landscape is the cost function we built in the last post. And instead of your feet feeling the slope, the model calculates it directly.
Connecting This Back to the Cost Function
Remember the cost function J(w, b) from the last post. If you tried every possible combination of w and b and plotted the cost for each one, you'd get a bowl shaped surface. Bad guesses for w and b sit high up on the sides of the bowl, where the cost is large. The best possible guess sits right at the bottom, where the cost is at its lowest.
Training a model is really just this. Starting somewhere on that bowl, usually a pretty bad, random spot, and walking downhill, step by step, until you reach the bottom.
The only thing we need to figure out is: at any given spot on this bowl, which direction is downhill?
Feeling the Slope, Without Feeling Anything
In real hills, you feel the slope with your feet. In gradient descent, the model calculates something called the gradient, which is just a fancy word for slope, at the current w and b.
The gradient tells you two things at once. It tells you which direction is uphill, and it tells you how steep the slope is at that point. Since we want to go downhill, not uphill, we move in the opposite direction of the gradient.
Here's the update rule that does this, written out for both w and b:
new w = old w − (learning rate × slope of the cost with respect to w)
new b = old b − (learning rate × slope of the cost with respect to b)
Let's slow down and unpack every piece of this in plain words.
The slope of the cost with respect to w tells us, roughly, "if I nudge w up a little, does the cost go up or down, and by how much."
A positive slope means increasing w would make things worse, so we should decrease w instead.
A negative slope means increasing w would actually help, so we should increase it.
Subtracting the slope automatically does the right thing in both cases, without us needing to think about which direction to move in by hand.
The learning rate, usually written as the Greek letter alpha, controls how big a step we take each time. This is the one number in the whole formula that we choose ourselves, rather than something the model calculates.
Choosing How Big a Step to Take
This is where our earlier tea stall example fits perfectly.
Remember the stall owner from the first post in this series. Every day, customers told him whether the tea was too sweet or not sweet enough, and he adjusted the sugar he used the next time. Now imagine two extreme versions of him.
Version one hears "too sweet" and immediately dumps in a huge amount of extra sugar the next time, or panics and removes almost all of it. Every single day, the tea swings wildly between way too sweet and way too plain, and it never actually settles anywhere close to right. This is what happens when the learning rate is too large. The steps are so big that instead of settling near the bottom of the bowl, the model keeps overshooting it, sometimes even ending up worse off than where it started.
Version two hears "too sweet" and adjusts the sugar by an amount so tiny it's barely noticeable, one grain at a time. Technically, this is moving in the right direction. But it would take him months, maybe years, to actually reach the right amount of sweetness this way. This is what happens when the learning rate is too small. The model does eventually get closer to the answer, but it takes an enormous number of steps to get there, wasting time and computing power along the way.
The real stall owner, without ever thinking about it in these terms, found a middle ground. Noticeable adjustments, but not wild swings. That middle ground is exactly what a well chosen learning rate looks like.
Let's Actually Work Through One Step, By Hand
Let's go back to our familiar delivery time table.
| Distance (km) | Actual Delivery Time (min) |
|---|---|
| 1 | 12 |
| 2 | 17 |
| 3 | 22 |
| 4 | 27 |
| 5 | 32 |
We already know the best answer is w = 5, b = 7. But let's pretend, once more, that we don't know this, and start from the worst possible guess. Let's start completely blindfolded, at w = 0 and b = 0.
With w = 0 and b = 0, every single prediction comes out to 0, no matter the distance. So our errors, calculated as prediction minus actual, are simply the negative of each actual value.
| Distance | Prediction | Actual | Error (prediction minus actual) |
|---|---|---|---|
| 1 | 0 | 12 | -12 |
| 2 | 0 | 17 | -17 |
| 3 | 0 | 22 | -22 |
| 4 | 0 | 27 | -27 |
| 5 | 0 | 32 | -32 |
The slope of the cost with respect to w is calculated as the average of each error multiplied by its corresponding distance:
slope for w = (1/5) × [(-12 × 1) + (-17 × 2) + (-22 × 3) + (-27 × 4) + (-32 × 5)]
slope for w = (1/5) × [-12 - 34 - 66 - 108 - 160]
slope for w = (1/5) × (-380)
slope for w = -76
The slope of the cost with respect to b is simply the average of the errors themselves, no distance multiplied in:
slope for b = (1/5) × [-12 - 17 - 22 - 27 - 32]
slope for b = (1/5) × (-110)
slope for b = -22
Both slopes came out negative, which tells us both w and b should increase. That matches what we'd expect, since our guess of 0 and 0 is far too low compared to the correct answer of 5 and 7.
Now let's pick a learning rate. We'll use alpha equals 0.01, a fairly small, safe value. Applying the update rule:
new w = 0 - (0.01 × -76) = 0 + 0.76 = 0.76
new b = 0 - (0.01 × -22) = 0 + 0.22 = 0.22
After just one single step, our guess moved from w = 0, b = 0 to w = 0.76, b = 0.22. Still far from the correct w = 5, b = 7, but moving in exactly the right direction.
This is the part that might surprise you. That's the entire algorithm. Compute the slope, take a small step against it, and repeat. Do this once, and you barely move. Do this a thousand times, and you land almost exactly on w = 5, b = 7, without ever eyeballing a table again.
Writing This as Code
Let's turn this into a function that does the update we just did by hand, and then repeats it automatically, many times.
def gradient_descent(x_list, y_list, w, b, learning_rate, iterations):
m = len(x_list)
for _ in range(iterations):
slope_w = 0
slope_b = 0
for i in range(m):
prediction = w * x_list[i] + b
error = prediction - y_list[i]
slope_w += error * x_list[i]
slope_b += error
slope_w = slope_w / m
slope_b = slope_b / m
w = w - learning_rate * slope_w
b = b - learning_rate * slope_b
return w, b
distances = [1, 2, 3, 4, 5]
actual_times = [12, 17, 22, 27, 32]
final_w, final_b = gradient_descent(distances, actual_times, w=0, b=0, learning_rate=0.01, iterations=10000)
print("Final w:", final_w)
print("Final b:", final_b)
# Final w: close to 5
# Final b: close to 7
Notice we never told this function that the answer was w = 5 and b = 7. It started at 0 and 0, completely blind, and found its own way there, purely by repeating the same small correction over and over.
Here's the same thing, vectorized with NumPy, since real datasets will have far more than five rows:
import numpy as np
def gradient_descent_vectorized(x, y, w, b, learning_rate, iterations):
m = len(x)
for _ in range(iterations):
predictions = w * x + b
errors = predictions - y
slope_w = np.sum(errors * x) / m
slope_b = np.sum(errors) / m
w = w - learning_rate * slope_w
b = b - learning_rate * slope_b
return w, b
distances_np = np.array([1, 2, 3, 4, 5])
actual_times_np = np.array([12, 17, 22, 27, 32])
final_w, final_b = gradient_descent_vectorized(distances_np, actual_times_np, w=0, b=0, learning_rate=0.01, iterations=10000)
print("Final w:", final_w)
print("Final b:", final_b)
Both versions do the exact same thing. Start with a bad guess. Look at the slope. Take a small step against it. Repeat, over and over, until the guess settles near the bottom of the bowl.
What We've Actually Built
Take a moment to notice what just happened across these last three posts. In the second post, we could only predict an outcome if a human had already handed us the right w and b. In this post, the model finds those numbers entirely on its own, using nothing but data and this one repeated idea of stepping downhill.
This combination, a prediction formula, a cost function to measure wrongness, and gradient descent to fix that wrongness automatically, is the actual engine behind linear regression. Every more advanced model you'll meet later in this series, no matter how complicated it sounds, is built on some variation of this exact same loop.
So far, we've only worked with one input feature, distance. Real problems are rarely that simple. A delivery time doesn't just depend on distance. It depends on the number of items in the order, the time of day, how busy the restaurant is, and plenty more. The next post is about extending everything we just built to handle more than one input at a time.
The full code from this post is up on GitHub here: https://github.com/tharun-pandya/Machine-Learning-Blog-Series/blob/main/Gradient_Descent.ipynb
See you in the next one.
