Logistic Regression: Drawing the Line Between Yes and No

Introduction: When Yes or No Matters

“CONGRATULATIONS!!! You have WON a FREE iPhone! CLICK HERE NOW!!!”

You must have seen this before and you instantly know what to do: SPAM. Delete.

Now, you do not even have to delete it anymore. Gmail, Outlook, and other email services automatically filter spam with impressive accuracy. The junk folder fills up while your inbox stays clean.

But how do computers learn to make that decision?

Unlike humans who recognize spam patterns intuitively, computers need to be taught. They need a model that can look at an email and answer: spam or not spam?

This is different from our last example. We are not predicting a continuous value like ice cream sales. We need a binary decision — yes or no.

That is what logistic regression does.

It takes the principles we have learned — gradients, optimization, gradient descent — and adapts them for classification instead of prediction.

Why Linear Regression Fails for Classification

In our previous blog, we used linear regression to predict continuous values — ice cream sales based on temperature. The model found the best-fit line, and the output could be any number: 250 scoops, 487 scoops, 631 scoops.

But classification is fundamentally different.

Let us try using linear regression for spam detection and see what breaks.

Problem 1: Outputs Are Unbounded

Linear regression gives us: ŷ = wx + b

This can output any real number — negative infinity to positive infinity.

But for spam classification, we need to assign labels:

• Spam = 1
• Not Spam = 0

We plot the data:

The problem is obvious: No straight line does justice to this data.

The line might predict:

• 0.3 for 2 exclamation marks (what does 0.3 spam mean?)
• 1.2 for 8 exclamation marks (can’t be more than 100% spam!)
• -0.1 for 0 exclamation marks (negative spam?)

We are trying to fit a continuous line to discrete categories. It does not work.

What we really need: A way to map any input to a probability between 0 and 1, not a line that shoots off to infinity.

Problem 2: MSE Does Not Fit Classification

Even if we force linear regression outputs between 0 and 1, the loss function is wrong.

Mean Squared Error (MSE) measures how far predictions are from actual values:

L = (predicted — actual)²

For regression, this makes sense:

• Predicting 450 when actual is 500 → error = 50
• Predicting 400 when actual is 500 → error = 100 (worse)

But for classification with labels 0 and 1:

• Predicting 0.1 when actual is 1 → it is wrong
• Predicting 0.2 when actual is 1 → also wrong

The difference between 0.1 and 0.2 does not matter much. What matters is: did you get the category right or wrong?

We need a loss function that:

• Heavily penalizes confident wrong predictions (predicting 0.9 when actual is 0)
• Barely penalizes predictions close to the boundary (0.4 or 0.6)

MSE treats all errors the same way. That is not what we want.

Problem 3: Sensitive to Outliers

Linear regression fits a line by minimizing squared errors across all points. This makes it sensitive to outliers.

Imagine spam emails with extreme feature values — 100 exclamation marks, 50 links. These outliers pull the line disproportionately.

But for classification, we just need a decision boundary — a line that separates spam from not spam. Extreme examples within each category should not affect the boundary much.

A spam email with 10 exclamation marks and one with 100 exclamation marks are both spam. The boundary should stay put.

What We Need Instead

We need a model that:

1. Outputs probabilities (0 to 1 range)
2. Uses a classification-friendly loss function (penalizes wrong categories, not distances)
3. Focuses on the boundary (not pulled by outliers)

That is what logistic regression provides.

Let us see how.

The Sigmoid Function: Squashing to Probabilities

We saw that linear regression fails for classification because it outputs unbounded values. We need something that:

1. Takes any input (negative infinity to positive infinity)
2. Outputs a probability (always between 0 and 1)
3. Creates a smooth decision boundary

Enters the sigmoid function. The sigmoid function is defined as:

Now ŷ represents the probability that an email is spam, always between 0 and 1.

How It Works

Let us understand what this function does by examining extreme cases:

Let us understand what this function does by examining extreme cases:

Case 1: When z is very negative (z → -∞)

Say z = -100:

• e⁻ᶻ = e⁻⁽⁻¹⁰⁰⁾ = e¹⁰⁰ → very large number
• Denominator: 1 + e¹⁰⁰ → huge
• Result: σ(z) = 1/(huge number) → approaches 0

Case 2: When z is very positive (z → +∞)

Say z = 100:

• e⁻ᶻ = e⁻¹⁰⁰ → very tiny (almost 0)
• Denominator: 1 + 0 → approaches 1
• Result: σ(z) = 1/1 → approaches 1

Case 3: When z = 0

• e⁰ = 1
• Denominator: 1 + 1 = 2
• Result: σ(0) = 1/2 = 0.5

No matter what z is, σ(z) is always between 0 and 1. Perfect for probabilities!

The S-Curve

Visually, the sigmoid function creates an S-shaped curve:

Sigmoid Function

Key properties:

• Left tail: As z → -∞, probability → 0 (definitely not spam)
• Center: At z = 0, probability = 0.5 (uncertain, boundary)
• Right tail: As z → +∞, probability → 1 (definitely spam)
• Smooth transition: No sudden jumps, gradual change

The sigmoid function transforms our linear model into a probability estimator. Now we have a proper classification model.

But how do we train it? How do we find the best w and b?

That requires a new loss function. Let us see what that is.

The Loss Function: Binary Cross-Entropy

We have our model—sigmoid transforms outputs to probabilities. Now we need a loss function to measure how wrong our predictions are.

We cannot use MSE. For classification, predicting 0.1 or 0.2 when the actual is 1 are both wrong—we got the category incorrect. We need a loss function designed for probabilities.

Enter Binary Cross-Entropy (Log Loss).

How It Works

The clever part: only one term matters because y is either 0 or 1.

When y = 1 (actual is spam):

• Second term becomes 0
• Loss = -log(ŷ)

Examples:

• ŷ = 0.9 (confident correct) → Loss ≈ 0.10 (low)
• ŷ = 0.5 (uncertain) → Loss ≈ 0.69 (medium)
• ŷ = 0.1 (confident wrong) → Loss ≈ 2.30 (high)

When y = 0 (actual is not spam):

• First term becomes 0
• Loss = -log(1-ŷ)

Examples:

• ŷ = 0.1 (confident correct) → Loss ≈ 0.10 (low)
• ŷ = 0.5 (uncertain) → Loss ≈ 0.69 (medium)
• ŷ = 0.9 (confident wrong) → Loss ≈ 2.30 (high)

Why Logarithms?

Logarithms heavily penalize confident wrong predictions:

• Predicting 0.9 when actual is 1: small loss
• Predicting 0.1 when actual is 1: 23× larger loss

Being confidently wrong hurts more than being uncertain. This pushes the model toward better decisions.

Our goal is to minimize this loss by finding the best w and b using gradient descent. Let us derive the gradients in next section.

Training: Gradient Descent with Sigmoid

We have all the pieces now:

• Model: ŷ = σ(wx + b) outputs probabilities
• Loss function: Binary cross-entropy measures prediction errors
• Goal: Find w and b that minimize loss

Time to train the model using gradient descent.

The Gradients

Just like linear regression, we need to compute how the loss changes with respect to our parameters:

Where ŷᵢ = σ(wxᵢ + b).

Notice something? These look identical to linear regression gradients! The only difference is that ŷ now comes from the sigmoid function instead of a direct linear output.

This elegant result comes from the mathematics of combining sigmoid with binary cross-entropy — they are designed to work together.

The Algorithm

Surprisingly enough, the steps remain same as here, only underlying loss function has changed.

Step 1: Initialize parameters

• w = 0 (or random small value)
• b = 0 (or random small value)

Step 2: For each iteration:

1. Compute predictions, using sigmoid function.
2. Calculate loss.
3. Compute Gradients
4. Update Parameters
5. Check convergence

• If loss stops decreasing significantly, stop
• Otherwise, repeat from step 2

Same Process, Different Model

If you understood gradient descent for linear regression, you understand it for logistic regression. The algorithm is identical:

Linear Regression:

• Model: ŷ = wx + b
• Loss: Mean Squared Error
• Gradient descent updates w and b

Logistic Regression:

• Model: ŷ = σ(wx + b)
• Loss: Binary Cross-Entropy
• Gradient descent updates w and b

The core optimization process remains unchanged. Only the model and loss function differ.

Learning Rate

Just like before, choose α carefully:

• Too large: overshoots, does not converge
• Too small: painfully slow training
• Typical values: 0.01, 0.001, 0.0001

Start with 0.01 and adjust based on whether loss decreases smoothly.

Next, let us implement this in code and watch it classify spam emails.

Conclusion

Linear regression predicts continuous values. Logistic regression classifies categories.

Same gradient descent algorithm. Different model (sigmoid), different loss (cross-entropy).

The pattern is clear:

• Choose the right model for your problem
• Choose the right loss function to measure errors
• Use gradient descent to optimize

This generalizes. Different problems need different models and losses. But the optimization process — gradient descent — remains constant.

What is next?

Neural networks. Stack multiple layers of transformations. Learn complex, non-linear patterns. Same calculus foundation, larger scale.

The math we have built does not change. It just gets applied to bigger, more powerful models.