1.3 - Multioutput Linear Regression

As the final step in developing the model, rather than modelling the relationship between multiple inputs and a single output, we will have multiple outputs at the same time. This problem is known as Multioutput Linear Regression.

Figure 1:Multioutput perceptron with multiple inputs and multiple outputs.

We can think of this model as a single neuron that takes multiple inputs and returns multiple outputs; we can also think of it as a dense layer containing multiple neurons.

Figure 2:Multioutput perceptron as a dense layer.

Both interpretations are valid; the idea of having a single neuron that returns multiple outputs is a useful concept for the following section Classification, when we introduce the activation function. The concept of grouping neurons into a single layer is useful for chapter Multilayer Perceptron, when we introduce multiple dense layers.

We assume that the true unknown function $f$ that maps the relationship between the input and output is:

\mathbf{y} = f\left( \mathbf{x} \right) + \epsilon

(1)

where $\epsilon$ is an intrinsic noise independent of $\mathbf{x}$ , normally $\epsilon \sim \mathcal{N}\left(0, \sigma^{2} \right)$ . Note that the target $\mathbf{y}$ is now a vector.

The goal of multioutput linear regression is estimate to $f$ by a linear approximation $\hat{f}$ such that:

\hat{\mathbf{y}} = \hat{f}(\mathbf{x}) \approx \mathbf{y}

(2)

This means that we want to estimate multiple output features from multiple input features.

Purpose of this Notebook:

Create a dataset for multioutput linear regression task
Create our own Multioutput Perceptron class from scratch
Calculate the gradients from scratch
Implement gradient descent from scratch
Train our Perceptron
Compare our Perceptron to PyTorch’s built-in implementation

Setup¶

print('Start package installation...')

Start package installation...

%%capture
%pip install torch
%pip install scikit-learn

print('Packages installed successfully!')

Packages installed successfully!

import torch
from torch import nn

from platform import python_version
python_version(), torch.__version__

('3.14.7', '2.12.0+cpu')

# Set seeds for reproducibility
device = 'cpu'
torch.manual_seed(13)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(13)
    device = 'cuda'
    import numpy as np; np.random.seed(13)
import random; random.seed(13)
device

'cpu'

The add_to_class function is used to add new methods to a previously defined class; we do this to gradually enhance the class.

def add_to_class(Class):  
    """Register functions as methods in created class."""
    def wrapper(obj): setattr(Class, obj.__name__, obj)
    return wrapper

Dataset¶

Create Dataset¶

The dataset $\mathcal{D}$ consists of input-target pairs $(\mathbf{x}_i, \mathbf{y}_i)$ :

\mathcal{D} = \left\{(\mathbf{x}_{1}, \mathbf{y}_{1}), \cdots, (\mathbf{x}_{n}, \mathbf{y}_{n}) \right\}

(3)

where $\mathbf{x}_{i} \in \mathbb{R}^{d}$ denotes the $i$ -th input sample with $d$ input features and $\mathbf{y}_{i} \in \mathbb{R}^{c}$ its corresponding target value with $c$ output features.

The input data $\mathbf{X} \in \mathbb{R}^{n \times d}$ can be represented as a matrix:

\mathbf{X} = \begin{bmatrix} x_{11} & \cdots & x_{1d} \\ \vdots & \ddots & \vdots \\ x_{n1} & \cdots & x_{nd} \end{bmatrix}

(4)

where $n$ is the number of samples in the dataset.

The target data also can be represented as a matrix $\mathbf{Y} \in \mathbb{R}^{n \times c}$ :

\mathbf{Y} = \begin{bmatrix} y_{11} & \cdots & y_{1 c} \\ \vdots & \ddots & \vdots \\ y_{n1} & \cdots & y_{n c} \end{bmatrix}

(5)

from sklearn.datasets import make_regression
import random

N: int = 1_500  # number of samples
D: int = 5  # number of input features
C: int = 3  # number of output features

X, Y = make_regression(  # type: ignore
    n_samples=N,
    n_features=D,
    n_targets=C,
    n_informative=N - 1,
    bias=random.random(),
    noise=1
)

X = X.astype(np.float32)
Y = Y.astype(np.float32)

X.shape, Y.shape

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[7], line 17
     13     bias=random.random(),
     14     noise=1
     15 )
     16 
---> 17 X = X.astype(np.float32)
     18 Y = Y.astype(np.float32)
     19 
     20 X.shape, Y.shape

NameError: name 'np' is not defined

Split Dataset¶

from sklearn.model_selection import train_test_split

X_train, X_valid, Y_train, Y_valid = train_test_split(
    X, Y,
    test_size=0.2,
    random_state=42,
    shuffle=True,
)

X_train.shape, Y_train.shape

X_valid.shape, Y_valid.shape

Remark: We have omitted the use of the test set for the rest of this notebook, as our aim here is to understand how the Perceptron works, not to draw general conclusions about a synthetic dataset. We will use the validation set instead of the test set, as it serves the same purpose in this notebook.

Tensor Dataset¶

from torch.utils.data import TensorDataset

train_dataset = TensorDataset(
    torch.from_numpy(X_train),
    torch.from_numpy(Y_train)
)

Get the first train input sample:

train_dataset[0][0]

And get the first target sample:

train_dataset[0][1]

valid_dataset = TensorDataset(
    torch.from_numpy(X_valid),
    torch.from_numpy(Y_valid)
)

Data Loader¶

from torch.utils.data import DataLoader

BATCH_SIZE: int = 32

train_loader = DataLoader(
    train_dataset,
    batch_size=BATCH_SIZE,
    shuffle=False,  # we want to ensure determinism
    pin_memory=True,
)

valid_loader = DataLoader(
    valid_dataset,
    batch_size=BATCH_SIZE,
    shuffle=False,
    pin_memory=True,
)

Scratch Multioutput Perceptron¶

Bias and Weight¶

Our model $\hat{\mathbf{y}}$ has two trainable parameters $\mathbf{b, W}$ called bias and weight respectively

\begin{align} \mathbf{b} \in \mathbb{R}^{c} \\ \mathbf{W} \in \mathbb{R}^{d \times c} \end{align}

(6)

class MultioutputRegression:
    def __init__(self, n_features: int, out_features: int):
        self.b = torch.randn(out_features).to(device)
        self.w = torch.randn(n_features, out_features).to(device)

    def copy_params(self, torch_layer: nn.modules.linear.Linear):
        """
        Copy the parameters from a module.linear to this model.

        Args:
            torch_layer: Pytorch module from which to copy the parameters.
        """
        self.b.copy_(torch_layer.bias.detach())
        self.w.copy_(torch_layer.weight.T.detach())

Weighted Sum¶

The weight sum is a vector-matrix multiplication between the input and weight plus a vector summation using broadcasting mechanism:

\begin{align} \hat{\mathbf{y}}: \mathbb{R}^{d} &\to \mathbb{R}^{c} \\ \mathbf{x} &\mapsto \hat{\mathbf{y}} \mathbf{(x) = b + xW} \end{align}

(7)

Notice that the predicted output is a vector.

For vectorization, given a minibatch input $\mathbf{X} \in \mathbb{R}^{m \times d}$ of $m$ samples and $d$ input features, we can compute the weighted sum as:

\hat{\mathbf{Y}} = \mathbf{b} + \mathbf{XW}

(8)

where the predict matrix $\hat{\mathbf{Y}} \in \mathbb{R}^{m \times c}$ of $c$ output features.

Note: $n$ is the number of samples in the dataset, while $m$ is the number of samples in a minibatch.

The prediction for the $i$ -th sample and $j$ output feature is:

\hat{y}_{ij} = b_{j} + \sum_{k=1}^{d} x_{ik} w_{kj}

(9)

This formula will be useful for gradient descent.

@add_to_class(MultioutputRegression)
def predict(self, x: torch.Tensor) -> torch.Tensor:
    """
    Predict the output for input x

    Args:
        x: Input tensor of shape (m_samples, d_features).

    Returns:
        y_pred: Predicted output tensor of shape (m_samples, c_features).
    """
    return torch.matmul(x, self.w) + self.b

MSE¶

The MSE function as loss function $\mathcal{L}$ needs a little update from its predecessor for multioutput features:

\begin{align} \mathcal{L}: \mathbb{R}^{m \times c} &\to \mathbb{R}_{\geq 0} \\ \hat{\mathbf{Y}} &\mapsto \mathcal{L}(\hat{\mathbf{Y}}), \; \hat{\mathbf{Y}} \in \mathbb{R}^{m \times c} \end{align}

(10)

MSE is defined as:

\begin{align} \mathcal{L} (\mathbf{W, b}) &= \frac{1}{mc} \sum_{i=1}^{m} \sum_{j=1}^{c} \left( \hat{y}_{ij} - y_{ij} \right)^{2} \\ &= \frac{1}{mc} \sum_{i=1}^{m} \sum_{j=1}^{c} \left( \left(b_{j} + \mathbf{x}^{\top}_{i} \mathbf{w}_{:,j} \right) - y_{ij} \right)^{2} \\ &= \frac{1}{mc} \sum_{i=1}^{m} \sum_{j=1}^{c} \left( \left(b_{j} + \sum_{k=1}^{d} x_{ik} w_{kj} \right) - y_{ij} \right)^{2} \end{align}

(11)

where $\mathbf{x}^{\top}_{i}$ is the $i$ -th sample/row of $\mathbf{X}$ as a vector $\mathbb{R}^{d}$ , and $\mathbf{w}_{:,j}$ is the $j$ -th column of the weight matrix $\mathbf{W}$ as a vector $\mathbb{R}^{d}$ .

For vectorization, we can compute MSE as:

\mathcal{L}(\mathbf{W, b}) = \frac{1}{mc} \text{ sum } \left( \mathbf{e} \odot \mathbf{e} \right)

(12)

where $\mathbf{e} = \hat{\mathbf{Y}} - \mathbf{Y}$ .

Note: $\odot$ is called Hadamard product and performs element-wise product.

@add_to_class(MultioutputRegression)
def mse_loss(self, y_true: torch.Tensor, y_pred: torch.Tensor):
    """
    MSE loss function between target y_true and y_pred.

    Args:
        y_true: Target tensor of shape (m_samples, c_features).
        y_pred: Predicted tensor of shape (m_samples, c_features).

    Returns:
        loss: MSE loss between predictions and true values.
    """
    return ((y_pred - y_true) ** 2).mean().item()

@add_to_class(MultioutputRegression)
def evaluate(self, x: torch.Tensor, y_true: torch.Tensor):
    """
    Evaluate the model on input x and target y_true using MSE.

    Args:
        x: Input tensor of shape (m_samples, d_features).
        y_true: Target tensor of shape (m_samples, c_features).

    Returns:
        loss: MSE loss between predictions and true values.
    """
    y_pred = self.predict(x)
    return self.mse_loss(y_true, y_pred)

Gradients¶

Derivative of MSE with respect to bias:

\frac{\partial \mathcal{L}}{\partial b_{r}} = \sum_{p,q} \frac{\partial \mathcal{L}}{\partial \hat{y}_{pq}} \frac{\partial \hat{y}_{pq}}{\partial b_{r}}

(13)

and derivative of MSE with respect to weight:

\frac{\partial \mathcal{L}}{\partial w_{rs}} = \sum_{p,q} \frac{\partial \mathcal{L}}{\partial \hat{y}_{pq}} \frac{\partial \hat{y}_{pq}}{\partial w_{rs}}

(14)

where the shape of each derivative is:

\begin{align} \frac{\partial \mathcal{L}}{\partial \mathbf{b}} & \in \mathbb{R}^{c} \\ \frac{\partial \mathcal{L}}{\partial \mathbf{W}} & \in \mathbb{R}^{d \times c} \\ \frac{\partial \mathcal{L}}{\partial \hat{\mathbf{Y}}} & \in \mathbb{R}^{m \times c} \\ \frac{\partial \hat{\mathbf{Y}}}{\partial \mathbf{b}} & \in \mathbb{R}^{(m \times c) \times c} \\ \frac{\partial \hat{\mathbf{Y}}}{\partial \mathbf{W}} & \in \mathbb{R}^{(m \times c) \times (d \times c)} \end{align}

(15)

Note: Third- and fourth-order derivatives may seem complicated, but you’ll soon find that the Kronecker delta property reduces the order when we differentiate.

Remark: The way to calculate derivatives using mixed-layout is to multiply the order of the dependent variable by the order of the independent variable. Orders of 0, such as the loss function, are not included.

For example, let $\mathbf{y} \in \mathbb{R}^{a_{1}, \ldots, a_{n}}$ and $\mathbf{x} \in \mathbb{R}^{b_{1}, \ldots, b_{m}}$ , then:

\frac{\partial \mathbf{y}}{\partial \mathbf{x}} \in \mathbb{R}^{\left( a_{1}, \ldots, a_{n} \right) \times \left( b_{1}, \ldots, b_{m} \right)}

(16)

The use of brackets is purely for visual purposes and does not alter the order of the derivative.

MSE derivative¶

Derivative of MSE with respect to predicted data is:

\begin{align} \frac{\partial \mathcal{L}}{\partial \hat{y}_{pq}} &= \frac{\partial}{\partial \hat{y}_{pq}} \left( \frac{1}{mc} \sum_{i=1}^{m} \sum_{j=1}^{c} \left( \hat{y}_{ij} - y_{ij} \right)^{2} \right) \\ &= \frac{1}{mc} \sum_{i=1}^{m} \sum_{j=1}^{c} \frac{\partial}{\partial \hat{y}_{pq}} \left( \left( \hat{y}_{ij} - y_{ij} \right)^{2} \right) \\ &= \frac{2}{mc} \sum_{i=1}^{m} \sum_{j=1}^{c} \left( \hat{y}_{ij} - y_{ij} \right) \frac{\partial \hat{y}_{ij}}{\partial \hat{y}_{pq}} \\ &= \frac{2}{mc} \sum_{i=1}^{m} \sum_{j=1}^{c} \left( \hat{y}_{ij} - y_{ij} \right) \delta_{ip} \delta_{jq} \\ &= \frac{2}{mc} \sum_{i=1}^{m} \left( \hat{y}_{iq} - y_{iq} \right) \delta_{ip} \\ &= \frac{2}{mc} \left( \hat{y}_{pq} - y_{pq} \right) \\ \end{align}

(17)

for $p = 1, \ldots, m$ , and $q = 1, \ldots, c$ .

The vectorized form is:

\frac{\partial \mathcal{L}}{\partial \hat{\mathbf{Y}}} = \frac{2}{mc} \left( \hat{\mathbf{Y}} - \mathbf{Y} \right)

(18)

Weighted Sum Derivative¶

With Respect to Bias¶

\begin{align} \frac{\partial \hat{y}_{pq}}{\partial b_{r}} &= \frac{\partial}{\partial b_{r}} \left( b_{q} + \sum_{k=1}^{d} x_{pk} w_{kq} \right) \\ &= \frac{\partial b_{q}}{\partial b_{r}} \\ &= \delta_{qr} \end{align}

(19)

for $p = 1, \ldots, m$ , and $q, r = 1, \ldots, c$ .

With Respect to Weight¶

\begin{align} \frac{\partial \hat{y}_{pq}}{\partial w_{rs}} &= \frac{\partial}{\partial w_{rs}} \left( b_{q} + \sum_{k=1}^{d} x_{pk} w_{kq} \right) \\ &= x_{pk} \frac{\partial w_{kq}}{\partial w_{rs}} \\ &= x_{pk} \delta_{kr} \delta_{qs} \\ &= x_{pr} \delta_{qs} \end{align}

(20)

for $p = 1, \ldots, m$ , for $q, s = 1, \ldots, c$ , and $r = 1, \ldots, d$ .

Full Chain Rule¶

Derivative of MSE with respect to bias:

\begin{align} \frac{\partial \mathcal{L}}{\partial b_{r}} &= \sum_{p, q} {\color{Cyan} \frac{\partial \mathcal{L}}{\partial \hat{y}_{pq}}} {\color{Orange} \frac{\partial \hat{y}_{pq}}{\partial b_{r}}} \\ &= \sum_{p, q} {\color{Cyan} \frac{2}{m c} \left( \hat{y}_{pq} - y_{pq} \right)} {\color{Orange} \delta_{qr}} \\ &= \frac{2}{m c} \sum_{p, q} \left( \hat{y}_{pq} - y_{pq} \right) \delta_{qr} \\ &= \frac{2}{m c} \sum_{p} \left( \hat{y}_{pr} - y_{pr} \right) \\ &= \frac{2}{m c} \left( \hat{\mathbf{y}}_{:,r} - \mathbf{y}_{:,r} \right)^{\top} \mathbf{1} \end{align}

(21)

for $r = 1, \ldots, c$ , where $\mathbf{1} \in \mathbb{R}^{m}$ .

The vectorized form is:

\frac{\partial \mathcal{L}}{\partial \mathbf{b}} = \frac{2}{m c} \left( \hat{\mathbf{Y}} - \mathbf{Y} \right)^{\top} \mathbf{1}

(22)

Derivative of MSE with respect to weight:

\begin{align} \frac{\partial \mathcal{L}}{\partial w_{rs}} &= \sum_{p,q} {\color{Cyan} \frac{\partial \mathcal{L}}{\partial \hat{y}_{pq}}} {\color{Orange} \frac{\partial \hat{y}_{pq}}{\partial w_{rs}}} \\ &= \sum_{p, q} {\color{Cyan} \frac{2}{m c} \left( \hat{y}_{pq} - y_{pq} \right)} {\color{Orange} x_{pr} \delta_{qs}} \\ &= \frac{2}{m c} \sum_{p, q} \left( \hat{y}_{pq} - y_{pq} \right) x_{pr} \delta_{qs} \\ &= \frac{2}{m c} \sum_{p} \left( \hat{y}_{ps} - y_{ps} \right) x_{pr} \\ &= \frac{2}{m c} (\mathbf{x}_{:,r})^{\top} \left( \hat{\mathbf{y}}_{:,s} - \mathbf{y}_{:,s} \right) \\ \end{align}

(23)

for $r = 1, \ldots, d$ , and $s = 1, \ldots, c$ .

Note: Let $\mathbf{x}_{:,r}$ be the $r$ -th input features from all input in a minibatch:

\mathbf{x}_{:,r} = \begin{bmatrix} x_{1r} & \cdots & x_{mr} \end{bmatrix}^{\top}

(24)

and let $\hat{\mathbf{y}}_{:,s}, \mathbf{y}_{:,s}$ be the $s$ -th output feature from all the predicted and target minibatch respectively:

\begin{align} \hat{\mathbf{y}}_{:,s} = \begin{bmatrix} \hat{y}_{1s} & \cdots & \hat{y}_{ms} \end{bmatrix}^{\top} \\ \mathbf{y}_{:,s} = \begin{bmatrix} y_{1s} & \cdots & y_{ms} \end{bmatrix}^{\top} \end{align}

(25)

The vectorized form is:

\frac{\partial \mathcal{L}}{\partial \mathbf{W}} = \frac{2}{m c} \mathbf{X}^{\top} \left( \hat{\mathbf{Y}} - \mathbf{Y} \right)

(26)

Final Gradients¶

\nabla_{\mathbf{b}} \mathcal{L} = \frac{2}{m c} \left( \hat{\mathbf{Y}} - \mathbf{Y} \right)^{\top} \mathbf{1}

(27)

\nabla_{\mathbf{W}} \mathcal{L} = \frac{2}{m c} \mathbf{X}^{\top} \left( \hat{\mathbf{Y}} - \mathbf{Y} \right)

(28)

Parameters Update¶

\begin{align} \mathbf{b} &\leftarrow \mathbf{b} - \eta \nabla_{\mathbf{b}} \mathcal{L} \\ &= \mathbf{b} - \eta \left( \frac{2}{m c} \left( \hat{\mathbf{Y}} - \mathbf{Y} \right)^{\top} \mathbf{1} \right) \end{align}

(29)

\begin{align} \mathbf{W} &\leftarrow \mathbf{W} - \eta \nabla_{\mathbf{W}} \mathcal{L} \\ &= \mathbf{W} - \eta \left( \frac{2}{m c} \mathbf{X}^{\top} \left( \hat{\mathbf{Y}} - \mathbf{Y} \right) \right) \end{align}

(30)

where $\eta > 0$ is the learning rate.

@add_to_class(MultioutputRegression)
def update(self, x: torch.Tensor, y_true: torch.Tensor,
           y_pred: torch.Tensor, lr: float):
    """
    Update the model parameters.

    Args:
       x: Input tensor of shape (m_samples, d_features).
       y_true: Target tensor of shape (m_samples, c_features).
       y_pred: Predicted output tensor of shape (m_samples, c_features).
       lr: Learning rate. 
    """
    delta = 2 * (y_pred - y_true) / y_true.numel()
    self.b -= lr * delta.sum(dim=0)
    self.w -= lr * torch.matmul(x.T, delta)

Gradient Descent¶

@add_to_class(MultioutputRegression)
def fit(self, train_loader: DataLoader,
        epochs: int, lr: float,
        valid_loader: DataLoader):
    """
    Fit the model using gradient descent.

    Args:
        train_loader: Train dataloader.
        epochs: Number of epochs to fit.
        lr: Learning rate.
        valid_loader: Valid dataloader.
    """
    for epoch in range(epochs):
        # training epoch
        running_loss = 0.0
        for batch_x, batch_y in train_loader:
            # move only the current batch to VRAM
            batch_x = batch_x.to(device, non_blocking=True)
            batch_y = batch_y.to(device, non_blocking=True)

            # make predictions
            y_pred = self.predict(batch_x)

            running_loss += self.mse_loss(
                batch_y, y_pred
            )

            self.update(
                batch_x, batch_y,
                y_pred, lr
            )

        avg_loss = running_loss / len(train_loader)

        # validation epoch
        running_vloss = 0.0
        # disable gradient computation and reduce memory consumption.
        with torch.no_grad():
            for vbatch_x, vbatch_y in valid_loader:
                vbatch_x = vbatch_x.to(device, non_blocking=True)
                vbatch_y = vbatch_y.to(device, non_blocking=True)

                vy_pred = self.predict(vbatch_x)

                running_vloss += self.mse_loss(
                    vbatch_y, vy_pred
                )
        avg_loss_v = running_vloss / len(valid_loader)

        print(f'epoch: {epoch} - MSE: {avg_loss:.4f} - vMSE: {avg_loss_v:.4f}')

Scratch vs Torch.nn¶

Torch.nn model¶

class TorchLinearRegression(nn.Module):
    def __init__(self, d_features, c_out_features):
        super().__init__()
        self.layer = nn.Linear(
            d_features,
            c_out_features
        )
        self.loss = nn.MSELoss()

    def forward(self, x):
        return self.layer(x)

    @torch.inference_mode()
    def evaluate(self, x, y):
        self.eval()
        y_pred = self(x)
        return self.loss(y_pred, y).item()

    def fit(self, train_loader,
            epochs, lr,
            valid_loader):
        optimizer = torch.optim.SGD(self.parameters(), lr=lr)

        for epoch in range(epochs):
            self.train()  # use model.train() when training
            running_loss = 0.0  # train loss
            for batch_x, batch_y in train_loader:
                batch_x = batch_x.to(device, non_blocking=True)
                batch_y = batch_y.to(device, non_blocking=True)

                y_pred = self(batch_x)

                loss = self.loss(y_pred, batch_y)

                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

                running_loss += loss.item()
            avg_loss = running_loss / len(train_loader)

            # validation epoch
            self.eval()  # set the model to evaluation mode
            with torch.no_grad():
                running_vloss = 0.0
                for vbatch_x, vbatch_y in valid_loader:
                    vbatch_x = vbatch_x.to(device, non_blocking=True)
                    vbatch_y = vbatch_y.to(device, non_blocking=True)

                    vy_pred = self(vbatch_x)
                    vloss = self.loss(vy_pred, vbatch_y)
                    running_vloss += vloss.item()
            avg_loss_v = running_vloss / len(valid_loader)

            print(f'epoch: {epoch} - MSE: {avg_loss:.4f} - vMSE: {avg_loss_v:.4f}')

torch_model = TorchLinearRegression(D, C).to(device)

model = MultioutputRegression(D, C)

Eval¶

We use a $L_{2}$ norm between PyTorch model and our Scratch model as parameter discrepancy.

def l2(pred, true_): 
    if isinstance(pred, (float, int)) and isinstance(true_, (float, int)):
        return abs(pred - true_)
    return torch.linalg.norm(pred - true_).item()

Predictions pre-copy¶

Let compare the predictions of our model and PyTorch implementation using $L_{2}$ norm:

x_valid, y_valid = valid_dataset[:]
x_valid = x_valid.to(device, non_blocking=True)
y_valid = y_valid.to(device, non_blocking=True)

l2(
    model.predict(x_valid),
    torch_model(x_valid)
)

They differ considerably because each model has its own parameters initialized randomly and independently of the other model.

Copy Parameters¶

We copy the values of the PyTorch model parameters to our model.

model.copy_params(torch_model.layer)

Predictions post-copy¶

We measure the difference between the predictions of both models again.

l2(
    model.predict(x_valid),
    torch_model(x_valid)
)

Loss¶

l2(
    model.evaluate(x_valid, y_valid),
    torch_model.evaluate(x_valid, y_valid)
)

Training¶

We are going to train both models using the same hyperparameter values. If our model is well designed, then starting from the same parameters it should arrive at the same parameters’ values as the PyTorch model after training.

LR: float = 0.01  # learning rate
EPOCHS: int = 16  # number of epochs

model.fit(
    train_loader,
    EPOCHS, LR,
    valid_loader
)

torch_model.fit(
    train_loader,
    EPOCHS, LR,
    valid_loader
)

Predictions after Training¶

l2(
    model.predict(x_valid),
    torch_model(x_valid)
)

Bias Comparison¶

We directly measure the difference between the bias values of both models.

l2(
    model.b.clone(),
    torch_model.layer.bias.detach()
)

Weight Comparison¶

l2(
    model.w.clone(),
    torch_model.layer.weight.detach().T
)

Our model, built from scratch, works in the same way as the PyTorch’s built-in implementation according to $L_{2}$ . This notebook serves as the basis for the following section, where the weighted sum is not modified, but a new element activation function is added to it in order to model a new type of problem: Classification. That is why it is important to understand how and why the weighted sum works before adding further elements to our models.