1.1 - Simple Linear Regression - Inside Deep Learning

Before delving into deep architectures, the linear unit serves as our foundational primitive. A set of linear units operating in parallel constitutes a fully connected (dense) layer; stacking multiple dense layers with non-linear activation functions enables the construction of deep neural networks (DNNs).

Figure 1:Simple perceptron with one input and one output.

The objective of simple linear regression is to predict the target data $y$ based on the input data $x$ :

y = f(x) + \epsilon

(1)

where $f$ represents the unknown underlying function, and $\epsilon$ denotes irreducible noise independent of $x$ , conventionally distributed as $\epsilon \sim \mathcal{N}(0, \sigma^{2})$ .

Given that the true function $f$ is unknown to us, we can estimate $f$ by $\hat{f}$ assuming that $f$ is approximately linear:

\hat{y} = \hat{f}(x) \approx y

(2)

where $\hat{y}$ is the predicted response.

Purpose of this Notebook:

Synthesize a 1D linear regression dataset.
Implement a single-unit Perceptron from first principles.
Derive analytical gradients via the multivariable chain rule.
Implement vectorized mini-batch gradient descent.
Train the custom model and monitor convergence.
Benchmark and numerically verify equivalence against PyTorch’s nn.Linear.

Setup¶

print("Start package installation...")

Start package installation...

%%capture
%pip install torch
%pip install scikit-learn
%pip install matplotlib

print("Packages installed successfully!")

Packages installed successfully!

from platform import python_version

import torch
from torch import nn

python_version(), torch.__version__

('3.14.7', '2.12.0+cpu')

# Set seeds for reproducibility
import random

import numpy as np

SEED: int = 11

random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
device = "cpu"

if torch.cuda.is_available():
    torch.cuda.manual_seed_all(SEED)
    device = "cuda"

device

'cpu'

Dataset¶

Create Dataset¶

For our supervised task, we have a dataset denoted:

\mathcal{D} = \left\{ (x_{1}, y_{1}), \cdots, (x_{n}, y_{n}) \right\}

(3)

where $n$ is the number of samples in our dataset.

We model the data generating process under the standard assumption that sample pairs $(x_{1}, y_{1}), \cdots, (x_{n}, y_{n})$ are independent and identically distributed (i.i.d.) according to an unknown joint distribution $\mathcal{P_{X, Y}}$ .

The input data $\mathbf{x}$ can be represented as a vector:

\mathbf{x} = \begin{bmatrix} x_{1} \\ \vdots \\ x_{n} \end{bmatrix} \in \mathbb{R}^{n}

(4)

and the target data $\mathbf{y}$ can be also represented as a vector:

\mathbf{y} = \begin{bmatrix} y_{1} \\ \vdots \\ y_{n} \end{bmatrix} \in \mathbb{R}^{n}

(5)

import random

from sklearn.datasets import make_regression

N: int = 1_000  # number of samples

X, Y = make_regression(  # type: ignore
    n_samples=N,
    n_features=1,
    n_targets=1,
    bias=random.randint(-5, 5),  # random true bias
    noise=2,
)

X.shape, Y.shape

((1000, 1), (1000,))

Y = Y.reshape(-1, 1)  # add the axis of length 1

X = X.astype(np.float32)
Y = Y.astype(np.float32)

X.shape, Y.shape

((1000, 1), (1000, 1))

Split Dataset¶

We partition $\mathcal{D}$ into three pairwise-disjoint subsets:

Train set $\mathcal{D}_{\text{train}}$ : Used to optimize model parameters.
Validation set $\mathcal{D}_{\text{valid}}$ : Used to tune hyperparameters, monitor over-fitting, and perform model selection.
Test set $\mathcal{D}_{\text{test}}$ : Reserved strictly for evaluating unbiased generalization performance on unseen data.

Remark: $\mathcal{D}_{\text{train}}$ , $\mathcal{D}_{\text{valid}}$ and $\mathcal{D}_{\text{test}}$ are disjoint, $\mathcal{D}_{\text{train}} \cap \mathcal{D}_{\text{valid}} \cap \mathcal{D}_{\text{test}} = \varnothing$ .

Let $\mathbf{x}_{\text{train}}$ denote the training input data, $\mathbf{y}_{\text{train}}$ the training target data, $\mathbf{x}_{\text{valid}}$ the validation input data, $\mathbf{y}_{\text{valid}}$ the validation target data, $\mathbf{x}_{\text{test}}$ the test input data, and $\mathbf{y}_{\text{test}}$ the target data respectively.

Here we use train_test_split from sklearn to split our arrays into random train and a test subsets.

from sklearn.model_selection import train_test_split

X_train, X_valid, Y_train, Y_valid = train_test_split(
    X,
    Y,
    test_size=0.2,
    random_state=42,
    shuffle=True,
)

X_train.shape, Y_train.shape

((800, 1), (800, 1))

X_valid.shape, Y_valid.shape

((200, 1), (200, 1))

Remark: We have omitted the use of the test set for the rest of this notebook, as our aim here is to understand how the Perceptron works, not to draw general conclusions about a synthetic dataset. We will use the validation set instead of the test set, as it serves the same purpose in this notebook.

Tensor Dataset¶

We convert our NumPy arrays into PyTorch tensors and wrap them in a TensorDataset to allow for structured iteration.

from collections.abc import Sized

from torch.utils.data import TensorDataset

train_dataset = TensorDataset(torch.from_numpy(X_train), torch.from_numpy(Y_train))
train_dataset[0]  # get the first sample

(tensor([0.9430]), tensor([4.9075]))

valid_dataset = TensorDataset(torch.from_numpy(X_valid), torch.from_numpy(Y_valid))

Data Loader¶

from torch.utils.data import DataLoader

BATCH_SIZE: int = 32

train_loader = DataLoader(
    train_dataset,
    batch_size=BATCH_SIZE,
    shuffle=False,  # we want to ensure determinism
    pin_memory=True,
    drop_last=True,
)

len(train_loader), len(train_dataset) // BATCH_SIZE

(25, 25)

_ = next(iter(train_loader))
_[0].shape, _[1].shape

/home/runner/work/inside-deep-learning/inside-deep-learning/.venv/lib/python3.14/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
  super().__init__(loader)

(torch.Size([32, 1]), torch.Size([32, 1]))

valid_loader = DataLoader(
    valid_dataset,
    batch_size=BATCH_SIZE,
    shuffle=False,
    pin_memory=True,
    drop_last=False,
)

_ = next(iter(valid_loader))
_[0].shape, _[1].shape

(torch.Size([32, 1]), torch.Size([32, 1]))

Plot Valid Samples¶

To illustrate the relationship between the input and output, we plot only 1000 points.

from matplotlib import pyplot as plt


def plot_lr(
    x: np.ndarray | torch.Tensor,
    y: np.ndarray | torch.Tensor,
    model: SimpleLR | TorchLR | None = None,
) -> None:
    plt.scatter(x, y, marker=".", label="Valid data")

    if model is not None:
        input_ = torch.tensor([x.min(), x.max()]).to(device)
        pred = model(input_)
        plt.plot(input_.cpu(), pred.cpu(), "r-", label="Predicted")

    plt.grid(True)
    plt.xlabel("x")
    plt.ylabel("y")
    plt.legend()
    plt.show()

x_val, y_val = valid_dataset[:]
plot_lr(x_val, y_val)

Simple Perceptron from Scratch¶

# Empty Linear Regression class
class SimpleLR:
    b: torch.Tensor
    w: torch.Tensor

    def copy_params(self, torch_layer: nn.modules.linear.Linear) -> None:
        pass

    def __call__(self, x: torch.Tensor) -> torch.Tensor:
        return torch.tensor([])

    def mse_loss(self, y_true: torch.Tensor, y_pred: torch.Tensor) -> float:
        return -1.0

    def evaluate(self, x: torch.Tensor, y_true: torch.Tensor) -> float:
        return -1.0

    def update(
        self, x: torch.Tensor, y_true: torch.Tensor, y_pred: torch.Tensor, lr: float
    ) -> None:
        pass

    def fit(
        self,
        train_: torch.utils.data.dataloader.DataLoader,
        epochs: int,
        lr: float,
        valid_: torch.utils.data.dataloader.DataLoader,
    ) -> None:
        pass

Bias and Weight¶

Our model $\hat{y}$ has two trainable parameters $b, w \in \mathbb{R}$ , which are called bias and weight respectively.

def init_params(self: SimpleLR) -> None:
    self.b = torch.randn(()).to(device)  # scalar
    self.w = torch.randn(()).to(device)  # scalar


setattr(SimpleLR, "__init__", init_params)

Let’s add this method copy_params so we can copy the parameter values from a PyTorch class to our scratch class.

def copy_params(self: SimpleLR, torch_layer: nn.modules.linear.Linear) -> None:
    """
    Copy the parameters from a module.linear to this model.

    Args:
        torch_layer: Pytorch module from which to copy the parameters.
    """
    self.b.copy_(torch_layer.bias.detach()[0])
    self.w.copy_(torch_layer.weight.detach()[0, 0])


setattr(SimpleLR, "copy_params", copy_params)

Weighted Sum¶

We selected the weighted sum for $\hat{y}$ as a linear approximation of the true function $f$ :

\begin{align} \hat{y}: \mathbb{R} &\to \mathbb{R} \\ x &\mapsto \hat{y}(x) = b + wx \end{align}

(6)

where $x \in \mathbb{R}$ represents an arbitrary input feature (not necessarily from the training distribution).

To improve computational efficiency, we vectorized calculations in minibatches of data. Given a minibatch $\mathbf{x} \in \mathbb{R}^{m}$ of $m$ samples:

\begin{align} \hat{\mathbf{y}} &= b + w \mathbf{x} \\ &= b + w \begin{bmatrix} x_{1} \\ \vdots \\ x_{m} \end{bmatrix} \\ &= \begin{bmatrix} b + wx_{1} \\ \vdots \\ b + wx_{m} \end{bmatrix} \end{align}

(7)

Note: Let $\hat{\mathbf{y}} \in \mathbb{R}^{m}$ denote the minibatch predicted data of $m$ samples.

Remark: Adding a scalar bias $b \in \mathbb{R}$ to a feature vector $\mathbf{x} \in \mathbb{R}^{m}$ relies on tensor broadcasting semantics, implicitly expanding $b$ to $b \mathbf{1} \in \mathbb{R}^{m}$ .

def predict(self: SimpleLR, x: torch.Tensor) -> torch.Tensor:
    """
    Predict the output for input x.

    Args:
        x: Input tensor of shape (m_samples,).

    Returns:
        y_pred: Predicted output tensor of shape (m_samples,).
    """
    return self.b + self.w * x


setattr(SimpleLR, "__call__", predict)

We visualize the regression line generated by randomly initialized parameters prior to training:

dummy_model = SimpleLR()
plot_lr(x_val, y_val, dummy_model)

MSE¶

We need a loss function $\mathcal{L}$ to help guide the adjustment of our parameters during training. We will use Mean Squared Error (MSE) as loss function:

\begin{align} \mathcal{L}: \mathbb{R}^{m} &\to \mathbb{R}_{\geq 0} \\ \hat{\mathbf{y}} &\mapsto \mathcal{L}(\hat{\mathbf{y}}), \; \hat{\mathbf{y}} \in \mathbb{R}^{m} \end{align}

(8)

MSE is defined as:

\begin{align} \mathcal{L} (w, b) &= \frac{1}{m} \sum_{i=1}^{m} \left( \hat{y}_{i} - y_{i} \right)^{2} \\ &= \frac{1}{m} \sum_{i=1}^{m} \left( \left( b + wx_{i} \right) - y_{i} \right)^{2} \end{align}

(9)

or using a vectorized form:

\mathcal{L} (w, b) = \frac{1}{m} \left\| \hat{\mathbf{y}} - \mathbf{y} \right\|^{2}_{2}

(10)

where $\left\| \mathbf{a} \right\|_{2}$ is the Euclidean norm or is also called $\ell_{2}$ norm (L2 norm).

def mse_loss(self: SimpleLR, y_true: torch.Tensor, y_pred: torch.Tensor) -> float:
    """
    MSE loss function between target y_true and y_pred.

    Args:
        y_true: Target tensor of shape (m_samples,).
        y_pred: Predicted tensor of shape (m_samples,).

    Returns:
        loss: MSE loss between predictions and true values.
    """
    return ((y_pred - y_true) ** 2).mean().item()


setattr(SimpleLR, "mse_loss", mse_loss)

def evaluate(self: SimpleLR, x: torch.Tensor, y_true: torch.Tensor) -> float:
    """
    Evaluate the model on input x and target y_true using MSE.

    Args:
        x: Input tensor of shape (m_samples,).
        y_true: Target tensor of shape (m_samples,).

    Returns:
        loss: MSE loss between predictions and true values.
    """
    y_pred = self(x)
    return self.mse_loss(y_true, y_pred)


setattr(SimpleLR, "evaluate", evaluate)

Gradients¶

To make our model’s parameters update, it is necessary to compute derivatives.

First, determine the derivatives to be computed
Then, ascertain the shape of each derivative
Finally, compute the derivatives

Using chain rule, we can determine the derivatives we need. Gradient of MSE with respect to bias is:

\frac{\partial \mathcal{L}}{\partial b} = \sum_{p} \frac{\partial \mathcal{L}}{\partial \hat{y}_{p}} \frac{\partial \hat{y}_{p}}{\partial b}

(11)

Gradient of MSE with respect to weight is:

\frac{\partial \mathcal{L}}{\partial w} = \sum_{p} \frac{\partial \mathcal{L}}{\partial \hat{y}_{p}} \frac{\partial \hat{y}_{p}}{\partial w}

(12)

Having defined the necessary derivatives, we now compute their shapes:

\frac{\partial \mathcal{L}}{\partial b} \in \mathbb{R}, \frac{\partial \mathcal{L}}{\partial w} \in \mathbb{R}, \frac{\partial \mathcal{L}}{\partial \hat{\mathbf{y}}} \in \mathbb{R}^{m}, \frac{\partial \hat{\mathbf{y}}}{\partial b} \in \mathbb{R}^{m}, \frac{\partial \hat{\mathbf{y}}}{\partial w} \in \mathbb{R}^{m}

(13)

MSE Derivative¶

The derivative of MSE with respect to predicted data:

\begin{align} \frac{\partial \mathcal{L}}{\partial \hat{y}_{p}} &= \frac{\partial}{\partial \hat{y}_{p}} \left( \frac{1}{m} \sum_{i=1}^{m} \left(\hat{y}_{i} - y_{i} \right)^{2} \right) \\ &= \frac{1}{m} \sum_{i=1}^{m} \frac{\partial}{\partial \hat{y}_{p}} \left( \left(\hat{y}_{i} - y_{i} \right)^{2} \right) \\ &= \frac{2}{m} \sum_{i=1}^{m} \left(\hat{y}_{i} - y_{i} \right) \frac{\partial \hat{y}_{i}}{\partial \hat{y}_{p}} \\ &= \frac{2}{m} \sum_{i=1}^{m} \left(\hat{y}_{i} - y_{i} \right) \delta_{ip} \\ &= \frac{2}{m} \left(\hat{y}_{p} - y_{p} \right) \end{align}

(14)

for $p = 1, \ldots, m$ .

Remark: Kronecker delta $\delta_{ij}$ is defined as:

\delta_{ij} = \begin{cases} 1 & \text{if } i=j \\ 0 & \text{if } i \neq j \end{cases}

(15)

and for any tensor $\mathbf{a}$ , $\sum_{i} \mathbf{a}_{i} \delta_{ij} = \mathbf{a}_{j}$ .

The vectorized form is:

\frac{\partial \mathcal{L}}{\partial \hat{\mathbf{y}}} = \frac{2}{m} \left( \hat{\mathbf{y}} - \mathbf{y} \right)

(16)

Weighted Sum Derivative¶

The derivative of weighted sum with respect to bias is:

\begin{align} \frac{\partial \hat{y}_{p}}{\partial b} &= \frac{\partial}{\partial b} \left(b + w x_{p} \right) \\ &= 1 \end{align}

(17)

for all $p = 1, \ldots, m$ .

Then, the vectorized form is

\frac{\partial \hat{\mathbf{y}}}{\partial b} = \mathbf{1}

(18)

where $\mathbf{1} \in \mathbb{R}^{m}$ .

The derivative of weighted sum with respect to weight is:

\begin{align} \frac{\partial \hat{y}_{p}}{\partial w} &= \frac{\partial}{\partial w} \left( b + w x_{p} \right) \\ &= \frac{\partial}{\partial w} \left( w x_{p} \right) \\ &= x_{p} \end{align}

(19)

for all $p = 1, \ldots, m$ .

Then, the vectorized form is:

\frac{\partial \hat{\mathbf{y}}}{\partial w} = \mathbf{x}

(20)

Full Chain Rule¶

Derivative of MSE with respect to bias is:

\begin{align} \frac{\partial \mathcal{L}}{\partial b} &= \sum_{p}^{} {\color{Cyan} \frac{\partial \mathcal{L}}{\partial \hat{y}_{p}} } {\color{Orange} \frac{\partial \hat{y}_{p}}{\partial b} } \\ &= \sum_{p} {\color{Cyan} \frac{2}{m} \left(\hat{y}_{p} - y_{p} \right)} {\color{Orange} 1} \\ &= \frac{2}{m} \sum_{p} \left(\hat{y}_{p} - y_{p} \right) 1 \\ &= \frac{2}{m} \left( \hat{\mathbf{y}} - \mathbf{y} \right)^{\top} \mathbf{1} \end{align}

(21)

Derivative of MSE with respect to weight is:

\begin{align} \frac{\partial \mathcal{L}}{\partial w} &= \sum_{p} {\color{Cyan} \frac{\partial \mathcal{L}}{\partial \hat{y}_{p}}} {\color{Magenta} \frac{\partial \hat{y}_{p}}{\partial w}} \\ &= \sum_{p} {\color{Cyan} \frac{2}{m} \left(\hat{y}_{p} - y_{p} \right)} {\color{Magenta} x_{p}} \\ &= \frac{2}{m} \sum_{p} \left(\hat{y}_{p} - y_{p} \right) x_{p} \\ &= \frac{2}{m} \left( \hat{\mathbf{y}} - \mathbf{y} \right)^{\top} \mathbf{x} \end{align}

(22)

Final Gradients¶

\nabla_{b} \mathcal{L} = \frac{\partial L}{\partial b} = \frac{2}{m} \left( \hat{\mathbf{y}} - \mathbf{y} \right)^{\top} \mathbf{1}

(23)

\nabla_{w} \mathcal{L} = \frac{\partial L}{\partial w} = \frac{2}{m} \left( \hat{\mathbf{y}} - \mathbf{y} \right)^{\top} \mathbf{x}

(24)

Parameters Update¶

Now, let’s update the trainable parameters using gradient descent (GD) as follows:

\begin{align} b &\leftarrow b -\eta \nabla_{b} L \\ &= b -\eta \left( \frac{2}{m} (\hat{\mathbf{y}} - \mathbf{y})^{\top} \mathbf{1} \right) \end{align}

(25)

\begin{align} w &\leftarrow w -\eta \nabla_{w} L \\ &= w -\eta \left( \frac{2}{m} (\hat{\mathbf{y}} - \mathbf{y})^{\top} \mathbf{x} \right) \end{align}

(26)

where $\eta > 0$ is called learning rate.

@torch.inference_mode()
def update(
    self: SimpleLR,
    x: torch.Tensor,
    y_true: torch.Tensor,
    y_pred: torch.Tensor,
    lr: float,
) -> None:
    """
    Update the model parameters.

    Args:
       x: Input tensor of shape (m_samples,).
       y_true: Target tensor of shape (m_samples,).
       y_pred: Predicted output tensor of shape (m_samples,).
       lr: Learning rate.
    """
    delta = 2 * (y_pred - y_true) / len(y_true)
    self.b -= lr * delta.sum()
    self.w -= lr * torch.matmul(delta.T, x).squeeze()


setattr(SimpleLR, "update", update)

Gradient Descent¶

We will use minibatch gradient descent (minibatch GD) to adjust the parameters of our model:

\begin{array}{l} \textbf{Algorithm: mini-batch Gradient Descent} \\ \textbf{for } t = 1 \text{ to } T \textbf{ do} \\ \quad i \leftarrow 1 \\ \quad j \leftarrow \mathcal{B} \\ \quad \textbf{while } i \leq m \textbf{ do} \\ \quad \quad \mathbf{\theta} \leftarrow \text{update}(\mathbf{x}_{\text{train } i:j}, \mathbf{y}_{\text{train } i:j}; \mathbf{\theta}) \\ \quad \quad i \leftarrow i + \mathcal{B} \\ \quad \quad j \leftarrow j + \mathcal{B} \\ \textbf{end for} \end{array}

(27)

where:

$T$ is the number of epochs.
$\theta$ is an arbitrary model’s parameter, in our case are $b$ and $w$ .
$\mathcal{B}$ is the number of samples per minibatch.
$\mathbf{x}_{\text{train } i:j}$ and $\mathbf{y}_{\text{train } i:j}$ are the $i$ -th to $j$ -th train samples.

Note: $\eta, T, \mathcal{B}$ are called hyperparameters, because they are adjusted by the developer rather than the model.

Remark: We intentionally do not shuffle the training data at each epoch for the sake of reproducibility. However, in practice, it is recommended to shuffle the training data at each epoch to improve convergence.

To learn more about types of gradient descents, please see gradient descents.

def fit(
    self: SimpleLR,
    train_: DataLoader,
    epochs: int,
    lr: float,
    valid_: DataLoader,
) -> None:
    """
    Fit the model using gradient descent.

    Args:
        train_: Training dataloader with tensors of shape (m_samples,).
        epochs: Number of epochs to fit.
        lr: learning rate.
        valid_: Validation dataloader with tensors of shape (m_valid_samples,).
    """
    # disable lint error: expected "Sized"
    assert isinstance(train_.dataset, Sized)
    assert isinstance(valid_.dataset, Sized)

    for epoch in range(epochs):
        # training epoch
        running_loss = 0.0
        for batch_x, batch_y in train_:
            # move only the current batch to VRAM
            batch_x = batch_x.to(device, non_blocking=True)
            batch_y = batch_y.to(device, non_blocking=True)

            # make predictions
            y_pred = self(batch_x)

            running_loss += self.mse_loss(batch_y, y_pred) * batch_y.size(0)

            self.update(batch_x, batch_y, y_pred, lr)
        avg_loss = running_loss / len(train_.dataset)

        # validation epoch
        running_vloss = 0.0
        # disable gradient computation and reduce memory consumption.
        with torch.no_grad():
            for vbatch_x, vbatch_y in valid_:
                vbatch_x = vbatch_x.to(device, non_blocking=True)
                vbatch_y = vbatch_y.to(device, non_blocking=True)

                vy_pred = self(vbatch_x)

                running_vloss += self.mse_loss(vbatch_y, vy_pred) * vbatch_y.size(0)

        avg_loss_v = running_vloss / len(valid_.dataset)

        print(f"epoch: {epoch} - MSE: {avg_loss:.4f} - vMSE: {avg_loss_v:.4f}")


setattr(SimpleLR, "fit", fit)

Scratch vs Torch.nn¶

We will be implementing a model created with PyTorch’s pre-built classes for linear regression. This will allow us to compare our model from scratch with the PyTorch model.

PyTorch Module¶

from typing import cast

class TorchLR(nn.Module):
    def __init__(self, n_features: int) -> None:
        super().__init__()
        self.layer = nn.Linear(n_features, 1)
        self.loss = nn.MSELoss()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return cast(torch.Tensor, self.layer(x))

    @torch.inference_mode()
    def evaluate(
        self,
        x: torch.Tensor,
        y: torch.Tensor,
    ) -> float:
        self.eval()
        y_pred = self(x)
        return float(self.loss(y_pred, y).item())

    def fit(
        self,
        train_: DataLoader,
        epochs: int,
        lr: float,
        valid_: DataLoader,
    ) -> None:
        optimizer = torch.optim.SGD(self.parameters(), lr=lr)

        # disable lint error: expected "Sized"
        assert isinstance(train_.dataset, Sized)
        assert isinstance(valid_.dataset, Sized)

        for epoch in range(epochs):
            self.train()  # use model.train() when training
            running_loss = 0.0
            for batch_x, batch_y in train_:
                batch_x = batch_x.to(device, non_blocking=True).unsqueeze(-1)
                batch_y = batch_y.to(device, non_blocking=True).unsqueeze(-1)

                y_pred = self(batch_x)

                loss = self.loss(y_pred, batch_y)

                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

                running_loss += loss.item() * batch_y.size(0)

            avg_loss = running_loss / len(train_.dataset)

            self.eval()  # set the model to evaluation mode
            with torch.no_grad():
                running_vloss = 0.0
                for vbatch_x, vbatch_y in valid_:
                    vbatch_x = vbatch_x.to(device, non_blocking=True)
                    vbatch_y = vbatch_y.to(device, non_blocking=True)

                    vy_pred = self(vbatch_x)
                    vloss = self.loss(vy_pred, vbatch_y)
                    running_vloss += vloss.item() * vbatch_y.size(0)

            avg_loss_v = running_vloss / len(valid_.dataset)

            print(f"epoch: {epoch} - MSE: {avg_loss:.4f} - vMSE: {avg_loss_v:.4f}")

torch_model = TorchLR(1).to(device)

# Let initialize our model from our class
model = SimpleLR()

plot_lr(x_val, y_val, model)

Eval¶

We use a $L_{2}$ norm between PyTorch model and our Scratch model as parameter discrepancy.

def l2(
    pred: int | float | torch.Tensor,
    true_: int | float | torch.Tensor,
) -> float:
    if isinstance(pred, (float, int)) and isinstance(true_, (float, int)):
        return abs(pred - true_)
    return float(torch.linalg.norm(pred - true_).item())

Predictions pre-copy¶

x_val = x_val.to(device, non_blocking=True)
y_val = y_val.to(device, non_blocking=True)

We measure the prediction discrepancy between both models using the Euclidean $L_{2}$ distance:

l2(model(x_val), torch_model(x_val))

8.669129371643066

The predictions diverge significantly due to independent pseudo-random weight and bias initializations.

Copy Parameters¶

We copy the values of the PyTorch model parameters to our model.

model.copy_params(torch_model.layer)

Predictions post-copy¶

We measure the difference between the predictions of both models again.

l2(model(x_val), torch_model(x_val))

0.0

Discrepancies on the order of $\sim 10^{-7}$ fall within the expected machine epsilon ( $\epsilon_{\text{mach}}$ ) for IEEE 754 single-precision floating-point arithmetic (float32), confirming numerical identity up to hardware precision limits.

Loss¶

l2(model.evaluate(x_val, y_val), torch_model.evaluate(x_val, y_val))

0.0

Training¶

We are going to train both models using the same hyperparameter values. Given identical initial weights, mini-batch sequences, and learning rates, our analytical gradient updates should produce an optimization trajectory numerically identical to PyTorch’s autograd engine.

LR: float = 0.01  # learning rate
EPOCHS: int = 16  # number of epochs

Remark: The training set can be divided evenly into four mini-batches, with each mini-batch containing exactly the same number of examples. Usually, in each training iteration, the training set is shuffled before being divided into mini-batches, and any remaining examples are discarded during that iteration.

model.fit(train_loader, EPOCHS, LR, valid_loader)

epoch: 0 - MSE: 48.7781 - vMSE: 27.4725
epoch: 1 - MSE: 19.7109 - vMSE: 12.8196
epoch: 2 - MSE: 9.5582 - vMSE: 7.6048
epoch: 3 - MSE: 6.0183 - vMSE: 5.7289
epoch: 4 - MSE: 4.7881 - vMSE: 5.0425
epoch: 5 - MSE: 4.3629 - vMSE: 4.7845
epoch: 6 - MSE: 4.2174 - vMSE: 4.6837
epoch: 7 - MSE: 4.1684 - vMSE: 4.6421
epoch: 8 - MSE: 4.1525 - vMSE: 4.6237
epoch: 9 - MSE: 4.1476 - vMSE: 4.6150
epoch: 10 - MSE: 4.1464 - vMSE: 4.6106
epoch: 11 - MSE: 4.1462 - vMSE: 4.6083
epoch: 12 - MSE: 4.1462 - vMSE: 4.6070
epoch: 13 - MSE: 4.1464 - vMSE: 4.6062
epoch: 14 - MSE: 4.1465 - vMSE: 4.6058
epoch: 15 - MSE: 4.1465 - vMSE: 4.6055

/home/runner/work/inside-deep-learning/inside-deep-learning/.venv/lib/python3.14/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
  super().__init__(loader)

torch_model.fit(train_loader, EPOCHS, LR, valid_loader)

epoch: 0 - MSE: 48.7781 - vMSE: 27.4725
epoch: 1 - MSE: 19.7109 - vMSE: 12.8196
epoch: 2 - MSE: 9.5582 - vMSE: 7.6048
epoch: 3 - MSE: 6.0183 - vMSE: 5.7289
epoch: 4 - MSE: 4.7881 - vMSE: 5.0425
epoch: 5 - MSE: 4.3629 - vMSE: 4.7845
epoch: 6 - MSE: 4.2174 - vMSE: 4.6837
epoch: 7 - MSE: 4.1684 - vMSE: 4.6421
epoch: 8 - MSE: 4.1525 - vMSE: 4.6237
epoch: 9 - MSE: 4.1476 - vMSE: 4.6150
epoch: 10 - MSE: 4.1464 - vMSE: 4.6106
epoch: 11 - MSE: 4.1462 - vMSE: 4.6083
epoch: 12 - MSE: 4.1462 - vMSE: 4.6070
epoch: 13 - MSE: 4.1464 - vMSE: 4.6062
epoch: 14 - MSE: 4.1465 - vMSE: 4.6058
epoch: 15 - MSE: 4.1465 - vMSE: 4.6055

/home/runner/work/inside-deep-learning/inside-deep-learning/.venv/lib/python3.14/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
  super().__init__(loader)

Predictions after Training¶

l2(model(x_val), torch_model(x_val))

0.0

plot_lr(x_val.cpu(), y_val.cpu(), model)

Bias Comparison¶

We directly measure the difference between the bias values of both models.

l2(model.b.clone(), torch_model.layer.bias.detach()[0])

0.0

Weight Comparison¶

And measure the difference between the weight values of both models.

l2(model.w.clone(), torch_model.layer.weight.detach()[0, 0])

0.0

Conclusion¶

In this notebook, we dissected the simplest foundational block of deep learning: the single-variable linear perceptron trained via Ordinary Least Squares (OLS) under Mean Squared Error (MSE).

Key Takeaways:¶

First-Principles Derivation: By applying the multivariable chain rule alongside Kronecker delta identities ( $\delta_{ip}$ ), we derived exact analytical expressions for parameter gradients ( $\nabla_w \mathcal{L}$ and $\nabla_b \mathcal{L}$ ). We demonstrated that expressing these derivatives in vectorized form ( $\mathbf{x}^\top \mathbf{\delta}$ ) eliminates scalar loops and maps directly to hardware-accelerated tensor contractions.
Numerical Equivalence with Autodiff: Benchmarking our custom implementation against PyTorch’s native nn.Linear and automatic differentiation engine yielded parameter and prediction discrepancies on the order of 10^-7. This proves that our manual gradient updates match PyTorch’s backward graph computation within the bounds of IEEE 754 single-precision floating-point tolerance.
Optimization Dynamics: Mini-batch gradient descent provided stable, monotonically decreasing loss trajectories across both training and validation sets, verifying that our batch-averaged gradient scaling correctly stabilizes the step size regardless of batch cardinality.

Next Steps:¶

Multivariate Linear Regression (Chapter 1.2): Extending input features from a scalar $x \in \mathbb{R}$ to a $d$ -dimensional feature vector $\mathbf{x} \in \mathbb{R}^d$ , requiring weight vectors $\mathbf{w} \in \mathbb{R}^d$ and full matrix operations ( $XW$ ).