The Gradient: Your Compass in High-Dimensional Space
When One Variable Is Not Enough
In the previous blog, we saw how derivatives identify the rate of change at the minuscule level instead of just averaging it. But everything we covered had one variable. f(x). Simple curves. Single input.
Real life isn’t that simple.
Your weight is a function of calorie intake AND exercise. House prices depend on size AND location. A neural network’s performance depends on thousands of parameters changing simultaneously.
With multiple variables, how do we figure out how EACH one impacts the output? And more importantly, what’s the optimal way to adjust ALL of them together?
That’s where partial derivatives and gradients come in.
Modern ML models have hundreds, thousands, even billions of parameters. Partial derivatives and gradients pave the path to optimize all of them for best performance.
Let’s see what these terms mean and why they matter.
Partial Derivatives: One Variable at a Time
Imagine you’re hiking in a beautiful mountainous region. (Btw, I love mountains, which makes this example more fun to explain!)
You’re standing at the foot of a hill. Height = f(x, y), where:
- x = how far east you are
- y = how far north you are
There are multiple routes to the top. You could head east, or north, or some combination. Which way is steeper? Which way climbs faster?
That’s what partial derivatives answer.
Let’s use a simple mountain: f(x, y) = x² + y²
This creates a bowl shape, with the lowest point at the origin.
- ∂f/∂x tells you: “If I walk east (holding north position constant), how steep is it?”
- ∂f/∂y tells you: “If I walk north (holding east position constant), how steep is it?”
So essentially, a partial derivative keeps all other variables constant and measures how changing just ONE variable impacts the function.
Let’s see it in an example,
for above function, when we take partial derivative of function w.r.t x then y is treated as constant which means, y² is essentially constant and its derivative is 0. Same for the partial derivative of function w.r.t y.
If you are standing at a point (3,4) then partial derivatives at that point are:
- ∂f/∂x = 2(3) = 6 → slope going east is 6
- ∂f/∂y = 2(4) = 8 → slope going north is 8
The north direction is steeper than the east direction at this point!
Visualizing it:
The animation below shows what it means to keep one variable constant while changing the other.
When we slice the surface parallel to the x-axis (holding y constant), we see the slope in the x-direction. When we slice parallel to the y-axis (holding x constant), we see the slope in the y-direction.
The Gradient: Putting It All Together
Now we have partial derivatives — the slope in each direction. But that doesn’t give us the complete picture.
You know the slope going east (∂f/∂x = 6) and the slope going north (∂f/∂y = 8). But what if you want to climb the steepest way possible? Which direction should you actually walk?
Not just east. Not just north. Some optimal combination of both.
That’s what the gradient tells you.
The gradient vector (written as ∇f, pronounced ‘del f’) combines all partial derivatives into one vector:
∇f = [∂f/∂x, ∂f/∂y]
For our mountain f(x, y) = x² + y²:
∇f = [2x, 2y]
At point (3, 4):
∇f(3, 4) = [6, 8]
What does this vector tell you?
- Direction: The gradient points in the direction of steepest ascent (fastest climb uphill)
- Magnitude: The length of the vector tells you how steep that steepest path is
Think of it as an arrow painted on the ground saying: “Walk THIS way to climb fastest.”
Why not just use the larger partial derivative?
Because the steepest path isn’t necessarily aligned with the x or y axis!
At (3, 4), the gradient [6, 8] points northeast-ish. It’s combining both directions optimally — a bit east (6 units) and more north (8 units).
The gradient at every point:
The gradient isn’t just one vector. It’s a vector field — at every point on the mountain, there’s a gradient vector pointing uphill from that spot.
Key properties:
- Gradient points toward higher values (uphill)
- Larger gradient = steeper slope
- Gradient = [0, 0] means you’re at a flat point (could be top, bottom, or saddle)
From Mountains To Machine Learning:
Our mountain example had two variables (x, y). Neural networks have thousands, even millions.
In ML, your loss function L(w₁, w₂, …, wₙ) depends on all these parameters. The gradient ∇L = [∂L/∂w₁, ∂L/∂w₂, …, ∂L/∂wₙ] tells you which direction improves performance.
But there’s a twist: we don’t want to climb UP (increase loss). We want to go DOWN (decrease loss).
So we follow the gradient in the opposite direction — that’s called gradient descent, and we’ll explore it in detail in upcoming blogs.
For now, remember: the gradient tells you which direction changes the function fastest. And in ML, we use that to navigate toward better models.
Conclusion: From One to Many
In the previous blog, we learned about derivatives — measuring how a function changes with a single variable. That was our foundation.
Now we’ve scaled up.
Partial derivatives let us isolate each variable’s impact. Hold everything else constant, see how one change affects the output. ∂f/∂x for east, ∂f/∂y for north.
The gradient combines them all into one powerful tool — a vector pointing in the direction of steepest ascent. Not just “this variable matters” but “here’s the optimal way to combine ALL variables.”
From hiking a two-variable mountain to training neural networks with millions of parameters, the principle is the same. Calculate the partial derivatives. Combine them into a gradient. Follow that direction.
What’s next?
We know what gradients are and where they point. But how do we actually use them to find the best solution? Where does the gradient equal zero, and what does that tell us?
Next up: Maxima, Minima, and Saddle Points — using gradients to locate and identify optimal points.
The tools are in place. Next, we will learn how to use them.
