Gradient Descent: Walking Downhill to Find the Bottom
Introduction: From Theory to Practice
In the previous blog, we found minima by setting ∇f = 0 and checking the second derivative.
That doesn’t work for real machine learning.
When training a model, you have a loss function — a measure of how wrong your predictions are. Your goal: minimize the loss by adjusting the model’s parameters.
Try solving ∇L = 0 for a billion-parameter neural network. It’s mathematically intractable. And even if you could, you’d still face local minima and saddle points.
Different approach: don’t solve, just walk.
Instead of solving equations algebraically, gradient descent takes an iterative approach: start somewhere, compute the gradient, take a small step in the direction that reduces loss, and repeat.
No complex equations to solve. Just follow the gradient downhill, one step at a time, until you reach a minimum.
This simple algorithm trains everything — linear regression, neural networks, ChatGPT.
Let’s see how it works.
The Algorithm: Following the Slope Downhill
The Intuition
Imagine you’re standing on a foggy mountain. It’s so foggy you can’t see more than a few feet ahead. You definitely can’t see the bottom.
Your goal: reach the valley at the base of the mountain.
How do you get down?
You can’t see the destination, but you can feel the slope beneath your feet.
So here’s your strategy:
- Feel which direction slopes downward most steeply
- Take a step in that direction
- Stop and feel the slope again
- Take another step downward
- Repeat
Step by step, following the steepest descent at each point, you’ll eventually reach the valley — even though you never saw where it was.
That’s exactly how gradient descent works.
- The mountain = your loss function
- Your position = current parameter values
- The slope = the gradient
- Stepping downward = updating parameters to reduce loss
- The valley = the minimum loss (optimal parameters)
You don’t need to see the whole landscape. You just need to know which direction is downhill right now, take a step, and repeat.
The Math
The math is surprisingly simple.
You have a loss function L that measures how bad your model’s predictions are. The loss depends on your model’s parameters (weights).
For example, in a simple linear model:
- Model: ŷ = w₁x₁ + w₂x₂ + … + wₙxₙ + b
- Loss: L(w₁, w₂, …, wₙ, b) = measure of prediction errors
Your goal: find the parameter values (w₁, w₂, …, wₙ, b) that minimize L.
Step 1: Start somewhere
Pick random initial values for your parameters. Call this w₀.
Step 2: Compute the gradient
Calculate ∇L — the gradient of the loss function with respect to all parameters.
Remember: ∇L points in the direction of steepest ascent (uphill). We want to go downhill, so we’ll move in the opposite direction: -∇L.
Step 3: Take a step downhill
Update your parameters:
Where:
- w_old = current parameters
- ∇L(w_old) = gradient at current position
- α (alpha) = learning rate (step size)
- w_new = updated parameters
Step 4: Repeat
Go back to Step 2 with your new parameters. Keep iterating until the gradient is close to zero (or you’ve reached your iteration limit).
That’s it. Compute gradient, step opposite direction, repeat.
Learning Rate: The Step Size That Makes or Breaks It
What is α?
The learning rate (α) controls how big a step you take downhill at each iteration.
It’s one of the most critical choices in gradient descent. Get it wrong, and the algorithm either crawls painfully slow or never converges at all.
Three scenarios:
Too Large (α = 1.0)
You might think: “Bigger steps = faster convergence, right?”
Wrong.
With steps that are too large, you overshoot the minimum. You leap past the valley to the other side of the mountain. Then you leap back. Then overshoot again.
The algorithm oscillates — bouncing back and forth, never settling down. It may never converge.
Too Small (α = 0.001)
Small steps are safe — you’ll never overshoot. You’re guaranteed to go downhill.
But it takes forever.
Thousands, maybe millions of iterations to reach the minimum. Each tiny step barely moves you. Progress is painfully slow.
For large models with billions of parameters, this is computationally infeasible.
Just Right (α = 0.1)
The sweet spot: steps large enough to make steady progress, but small enough to avoid overshooting.
The algorithm converges efficiently — not too fast, not too slow.
Finding the right learning rate:
There’s no universal “correct” value. It depends on:
- The shape of your loss function
- The scale of your parameters
- The problem you’re solving
Common approach:
- Try different values: 0.001, 0.01, 0.1, 1.0
- Run gradient descent with each
- Pick the one that converges fastest without oscillating
This trial-and-error process is part of hyperparameter tuning — finding the settings that work best for your specific problem.
Modern ML libraries often use adaptive learning rates that automatically adjust α during training, but understanding the basic principle helps.
Challenges: When Gradient Descent Struggles
Gradient descent is powerful, but it’s not perfect. Real-world loss functions are messy, and several things can go wrong.
Local minima:
Good coverage of challenges! Just needs some refinement and corrections. Here’s a polished version:
Challenges: When Gradient Descent Struggles
Gradient descent is powerful, but it’s not perfect. Real-world loss functions are messy, and several things can go wrong.
Local Minima
Not all functions have one clean minimum. Loss functions with billions of parameters look like mountain ranges — full of peaks and valleys.
Gradient descent follows the slope and finds a valley. But is it the deepest valley? Maybe not. You might get stuck in a local minimum — lower than the surrounding area, but not the global best.
The algorithm stops (gradient ≈ 0), but you’re not at the optimal solution.
In practice, this is less of a problem than once thought. Many local minima in neural networks perform well enough.
Saddle points:
Remember these from the last blog? Gradient equals zero, but it’s neither max nor min. The function curves up one way, down another.
High-dimensional spaces are full of saddle points — far more common than local minima.
Gradient descent can stall because ∇L ≈ 0. Fortunately, small random fluctuations usually help it escape and continue downhill.
Plateaus:
Flat regions where the gradient is extremely small but not quite zero.
Progress slows to a crawl. Each step barely moves you. It’s hard to tell if you’re near the minimum or stuck on flat terrain.
Solution: Patience. Run more iterations. You’ll eventually cross the plateau and find steeper ground.
Wrong learning rate:
We covered this above, but it’s critical:
- Too large: overshoots and oscillates, never settles
- Too small: gets there eventually, but painfully slow
Finding the right balance takes experimentation.
Despite these challenges, gradient descent works remarkably well. Modern variants have tricks to handle these issues, but the core idea stays the same: follow the gradient downhill.
These problems are real, but manageable. The algorithm trains everything from simple linear models to ChatGPT.
Why This Matters: The Engine of Machine Learning
Gradient descent trains everything. Linear regression, neural networks, ChatGPT — all use the same core algorithm:
- Start with random parameters
- Compute loss (how wrong are predictions?)
- Compute gradient (how to improve?)
- Update: w_new = w_old — α · ∇L
- Repeat until loss stops decreasing
It’s a simple algorithm with universal application.
One parameter or billions — the principle is the same. Compute gradient, step downhill, repeat.
From calculus to AI:
We started with derivatives (measuring change). Then gradients (change across dimensions). Then optimization (finding minima). Now gradient descent brings it all together.
You don’t solve equations. You just walk downhill, one step at a time, following the gradient.
It’s not perfect. But gradient descent is remarkably robust. It works.
This simple idea — follow the slope downhill — powers modern AI.
What’s next?
Now that we have built the foundation, we will look into our first regression model in next blog.
