Linear Regression: Where Math Meets Data

Introduction: Predicting from Data

Earlier we covered derivatives, gradients, and gradient descent. Now let’s put them to work.

Consider the scenario:

You own an ice cream shop. Summer is here, and sales fluctuate wildly day to day.

Monday: 35°C, blazing sun. You sell 850 scoops. Run out of inventory by 4 PM.

Tuesday: 18°C, cloudy and cool. You sell 180 scoops. Tons of melted leftovers.

Wednesday’s forecast: 28°C. Question: How much ice cream should you stock?

If stocking too little then it means lost sales. Stocking too much leads to waste. You need a way to predict demand.

You have last month’s data — daily temperatures and sales. Can you use it to predict tomorrow’s sales based on the weather forecast?

That is what linear regression can help you do.

It finds the relationship between temperature (input) and ice cream sales (output). Specifically, it finds the best-fit line through your data points.

Once you have that line, predictions are simple:

• Weather forecast says 28°C → plug into the line → predicted sales
• The model has “learned” from past data

The approach:

1. Plot your data — temperature vs sales (scatter plot)

2. Find the best line — the one that fits the points closest

3. Make predictions — plug in tomorrow’s temperature, get expected sales

Behind the scenes? Gradient descent adjusts the line’s slope and position until the fit is optimal — minimizing prediction errors.

Let’s see how it works.

The Model and Loss Function

The idea behind linear regression is simple: given historical data, find the best-fit line.

This is something we do intuitively. When we see scattered data points, our brain naturally tries to imagine a line through them. Linear regression does this mathematically.

The Model: A Line Equation

A line can be represented as:

where:

• ŷ (y-hat) = predicted value (ice cream sales)
• w = weight/slope (how much sales change per degree)
• x = input (temperature in °C)
• b = bias/intercept (base sales at 0°C)

Example: If w = 20 and b = 50, then:

• At 10°C: ŷ = 20(10) + 50 = 250 scoops
• At 30°C: ŷ = 20(30) + 50 = 650 scoops

Our goal: Find the best values for w and b.

The Problem: No Perfect Line

Plot your data points — temperature on x-axis, sales on y-axis.

You will notice: no single line passes through all points perfectly. Data is messy. There is always some scatter.

So what do we do? Find the line that gets as close as possible to all points. The line that minimizes the total error across all data points.

Error = difference between predicted and actual sales

The Loss Function: Mean Squared Error (MSE)

We measure “closeness” with a loss function:

where:

• n = number of data points
• ŷᵢ = predicted sales for day i
• yᵢ = actual sales for day i
• Σ = sum across all data points

In plain English:

1. For each day, compute (predicted — actual)
2. Square it
3. Average across all days

Lower loss = better fit.

Why Square the Errors?

Reason 1: Avoid cancellation

Without squaring:

• Day 1: predicted 500, actual 600 → error = -100
• Day 2: predicted 400, actual 300 → error = +100
• Sum = 0 (errors cancel out!)

This makes it look like the predictions are perfect when they are not.

Squaring makes all errors positive: (-100)² = 10,000 and (+100)² = 10,000.

Reason 2: Penalize large errors more

Squaring amplifies bigger mistakes:

• Small error:2² = 4
• Large error: 10² = 100

This pushes the algorithm toward a line that avoids big misses, even if it means slightly larger errors elsewhere. We want consistent predictions, not wild swings.

In nutshell, our goal is: Find w and b that minimize L(w, b).

That is where gradient descent comes in. But first, we need to compute the gradients of L with respect to w and b.

Let us derive them next.

The Math: Computing Gradients and Training with Gradient Descent

Our goal: minimize the loss function.

Remember what helps us find minima? Gradient descent!

The gradient points in the direction of steepest ascent (uphill). If we move in the opposite direction and keep recalculating the gradient after each step, we will reach the minimum.

What are we minimizing? The loss function L(w, b).

With respect to what? The parameters w (slope) and b (intercept).

We need the rate of change of L as we adjust w and b. That means computing the partial derivatives: ∂L/∂w and ∂L/∂b.

Deriving the Gradients

We have the loss function:

Substitute the model ŷ = wx + b:

Partial derivative with respect to w:

Using the chain rule:

Partial derivative with respect to b:

These gradients tell us how much the loss changes when we adjust w or b.

The Gradient Descent Algorithm

Now we use these gradients to find the optimal w and b.

Step 1: Initialize

Start with random values:

• w = 0 (or any random number)
• b = 0 (or any random number)

Step 2: Compute gradients

Calculate ∂L/∂w and ∂L/∂b using current w and b values.

Step 3: Update parameters

Move in the opposite direction of the gradient:

where α (alpha) is the learning rate — controls step size.

Step 4: Compute loss

Calculate L with the new w and b values.

Step 5: Repeat

Go back to Step 2 and repeat until:

• Loss stops decreasing significantly
• Gradient ≈ 0
• Maximum iterations reached

Choosing the Learning Rate

Remember from the gradient descent blog:

• Too large: Overshoots, oscillates, never converges
• Too small: Painfully slow progress
• Just right: Smooth convergence

Typically, try values like 0.001, 0.01, 0.1 and see what works best.

Watching It Learn

Let us apply this to our ice cream data. We will start with random w and b, then watch gradient descent adjust them iteration by iteration.

The line will start in a random position, then gradually move and rotate until it fits the data well. The loss will decrease with each iteration.

After convergence, we have our best-fit line:

• w = slope (how much sales change per degree)
• b = intercept (base sales)

Now we can predict: given tomorrow’s temperature, plug it into ŷ = wx + b and get expected sales!

Building It: Code and Visualization

Let us see the implementation. The complete Jupyter notebook with detailed comments and additional visualizations is available on GitHub: linear-regression.ipynb

Here are the core functions:

Compute gradients

def compute_gradient(x, y, w, b):
    N = len(y)
    y_pred = w * x + b
    error = y_pred - y
    dw = (2/N) * np.dot(error, x)
    db = (2/N) * np.sum(error)
    return dw, db

Update parameters

def find_new_slope_intercept(w, b, alpha):
    dw, db = compute_gradient(temperatures, sales, w, b)
    w_new = w - alpha * dw
    b_new = b - alpha * db
    return w_new, b_new

Compute and minimize loss

w_init = 0.0
b_init = 0.0
alpha = 0.0001
num_iterations = 10000

def minimize_cost(temperatures, sales, w_init, b_init, alpha, num_iterations):
    w = w_init
    b = b_init
    x = temperatures
    y = sales

    for i in range(num_iterations):
        y_pred = w * x + b
        cost = (1/len(y)) * np.sum((y_pred - y) ** 2)

        print(f"Iteration {i+1}: Cost = {cost:.4f}, w = {w:.4f}, b = {b:.4f}")

        if(cost < 1e-6):
            print("Convergence reached.")
            print(f"Final parameters: w = {w:.4f}, b = {b:.4f}")
            break

        w, b = find_new_slope_intercept(w, b, alpha)

    return w, b

w_final, b_final = minimize_cost(temperatures, sales, w_init, b_init, alpha, num_iterations)

Note: We chose α = 0.0001 after testing different values. Too large (α ≥ 0.001) caused the gradients to explode. Too small (α < 0.00001) would take forever. This value gives stable, steady convergence.

Here is the best-fit line for the ice cream sales data:

With our model ready, we can now use this to make predictions, like below.

def predict_sales(temperature):
    return w_final * temperature + b_final

# Tomorrow's forecast: 25°C - Predicted Value: 574 scoops
predicted_sales = predict_sales(25)
print(f"Expected sales at 25°C: {predicted_sales:.0f} scoops")

Conclusion

From derivatives to gradients to optimization to gradient descent — now we have seen it all work together in a real model.

Linear regression learns from data. It finds patterns. It makes predictions. The math is not abstract anymore. It is practical.

The process:

• Model: ŷ = wx + b
• Loss: measure errors
• Gradients: compute how to improve
• Update: w and b get better
• Repeat: until convergence

Same process trains neural networks with billions of parameters. Different scale, same algorithm.

Coming next: Logistic regression — applying these tools to classification problems instead of prediction.