Optimization with Derivatives: Finding the Peak (and the Valley)
The Search for Optimal
In the previous blog, we learned that the gradient is like a compass — it points in the direction of steepest ascent. If you are climbing a mountain, the gradient tells you which way is up.
But here’s the question: where does it lead?
And more importantly: how do you know when you have arrived?
When training a machine learning model, you are not trying to climb to the top. You are trying to find the bottom — the point where your loss function is minimized, where your model performs best.
The gradient points the way. But you need to know when to stop. When have you reached the valley? When is the loss as low as it can go?
That’s what we will see in this blog.
Critical points — places where the gradient equals zero — are the destinations. But not all destinations are equal. Some are peaks (maxima). Some are valleys (minima). And some are neither — they are saddle points, the trickiest case of all.
Let’s see how to identify where you have landed and what it means for optimization.
Critical Points: Where the Gradient Goes Silent
In simple terms, critical points are where the gradient equals zero.
We saw that the gradient is the collection of first-order partial derivatives — it tells you the slope in each direction. When the gradient is zero, there is no slope. The surface is flat at that point.
Let’s see this with an example.
In the previous blog, we used the function: f(x, y) = x² + y²
What type is it?
This is a minimum. If you recall the shape from the previous blog, it was a bowl — a paraboloid opening upward. The bottom of the bowl is at (0, 0), so a minimum at that point makes perfect sense.
At (0, 0), the function value is f(0, 0) = 0. Every other point has a higher value. You are at the lowest point.
Maxima and Minima: Peaks and Valleys
Above, I said the critical point (0, 0) is a minimum. But how did I know?
One way is visualization — we saw the bowl shape. But that’s not possible in higher dimensions. Machine learning models have loss functions with hundreds or thousands of variables. You can’t visualize a 1000-dimensional surface.
Fortunately, we don’t need to visualize. Mathematics tells us the answer.
The Second Derivative Test
Let’s start with a simpler single-variable example to build intuition.
Example 1: Finding a maximum of f(x) = 2 — x²
This tells us the function is concave down — like an upside-down bowl, or a mountain peak.
Therefore, x = 0 is a maximum.
Let’s see another example which has a minimum, f(x) = 2 + x²
Example 2: Finding a minimum
This tells us the function is concave up — like a bowl.
Therefore, x = 0 is a minimum.
The Rule (Single Variable):
At a critical point where f’(x) = 0:
- f’’(x) > 0 → Concave up (bowl) → Minimum
- f’’(x) < 0 → Concave down (dome) → Maximum
- f’’(x) = 0 → Test is inconclusive (could be anything)
What About Multiple Variables?
For functions with multiple variables like f(x, y), we use something called the Hessian matrix — a collection of all second-order partial derivatives.
Let’s see for the bowl function f(x, y) = x² + y²
Both second derivatives are positive, confirming that (0, 0) is indeed a minimum.
We won’t dive deep into Hessian analysis here, but the principle is the same: second derivatives tell you the curvature, and curvature tells you the type of critical point.
Saddle Points: The Deceptive Middle Ground
Not all critical points are clearly maxima or minima. Some are deceptive — they look flat (gradient = 0), but they’re neither a peak nor a valley.
These are called saddle points.
At a saddle point, the function curves upward in one direction and downward in another direction — all at the same point. It’s a maximum along one axis and a minimum along another.
Let’s see a classic example:
What does this mean?
At the point (0, 0):
- If you move in the x-direction, the function curves upward (like a bowl)
- If you move in the y-direction, the function curves downward (like an upside-down bowl)
The same point is a minimum in one direction and a maximum in another!
The surface looks like a horse saddle or a Pringles chip — hence the name. It curves up one way and down the other.
The tricky part:
The gradient at (0, 0) is zero — just like at a true maximum or minimum. If you only check that ∇f = 0, you can’t tell the difference. You need the second derivative test (Hessian) to distinguish saddle points from true optima.
Why saddle points matter in machine learning:
When training neural networks, the optimization algorithm can get “stuck” at saddle points. The gradient is zero, so gradient descent doesn’t know which way to go.
Worse yet, high-dimensional spaces are full of saddle points. A 1000-parameter model has exponentially more saddle points than local minima. Most critical points in deep learning are saddles, not true minima.
Modern optimization algorithms have tricks to escape saddle points, but they remain one of the challenges in training deep neural networks.
Why This Matters: The Optimization Landscape
Training a machine learning model is an optimization problem. You have a loss function L(w₁, w₂, …, wₙ) with thousands of parameters, and you need to minimize it.
The math is the same as we’ve been doing:
- Compute gradient ∇L
- Find where ∇L = 0
- Hope it’s a minimum!
But real loss functions are messy:
- Multiple local minima — many valleys to get trapped in
- Saddle points everywhere — especially in high dimensions
- Plateaus — flat regions that slow progress
- No clear path — narrow valleys, steep ridges
The problem:
When you find ∇L = 0, you don’t know if it’s:
- ✅ A good minimum
- ⚠️ A mediocre local minimum
- ❌ A saddle point (not even a minimum!)
In high-dimensional spaces, most critical points are saddle points, not minima. Neural networks could easily get stuck.
The solution:
Modern optimization algorithms (gradient descent variants) have tricks to handle this:
- Momentum to escape saddles
- Adaptive learning rates
- Noise to avoid getting stuck
And it works! We don’t need the perfect global minimum — a good local minimum is often excellent.
Conclusion: Identifying the Destination
The gradient is your compass. Critical points (where ∇f = 0) are your destinations.
But destinations come in three flavors:
- Minima — valleys (what we want)
- Maxima — peaks (what we avoid)
- Saddle points — neither (what tricks us)
The second derivative test tells you which type you’ve found. In machine learning with thousands of parameters, the Hessian matrix does the same job.
Real loss landscapes are messy — full of local minima and saddle points. Understanding critical points is the foundation for navigating them.
Coming next: How to actually find minima using gradient descent — following the gradient downhill, one step at a time.
From theory to practice. The optimization journey continues.
