What is hidden in torch.nn that is multiplied by the feature tensor of my data?

Question

Redoing a tutorial on Captum I have recreated its neural network, TitanicSimpleNNNModel, a simple architecture whereby the last layer performs a softmax operation and has 2 units, corresponding to the outputs of survived (1) or not survived (0). However the problem is that when I try to train it I get a dimension error on the matrices that the model tries to multiply: RuntimeError: mat1 and mat2 shapes cannot be multiplied (916x2 and 12x8). The first one corresponds in the training characteristics (train_features), which seems to be a cipher tensor, but I don't know which one the second one refers to. Indeed it seems that both are in an *input, and I was not able to find out which ones are in it.

Here you have my google colab notebook, but here is how I get the data, the model and the training:

# Initial imports
import numpy as np

import torch

from captum.attr import IntegratedGradients
from captum.attr import LayerConductance
from captum.attr import NeuronConductance

import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

from scipy import stats
import pandas as pd

# DATOS

# Download dataset from: https://biostat.app.vumc.org/wiki/pub/Main/DataSets/titanic3.csv
# Update path to dataset here.
! wget https://biostat.app.vumc.org/wiki/pub/Main/DataSets/titanic3.csv
dataset_path = "titanic3.csv"

# Read dataset from csv file.
titanic_data = pd.read_csv(dataset_path)

titanic_data = pd.concat([titanic_data,
                          pd.get_dummies(titanic_data['sex']),
                          pd.get_dummies(titanic_data['embarked'],prefix="embark"),
                          pd.get_dummies(titanic_data['pclass'],prefix="class")], axis=1)
titanic_data["age"] = titanic_data["age"].fillna(titanic_data["age"].mean())
titanic_data["fare"] = titanic_data["fare"].fillna(titanic_data["fare"].mean())
titanic_data = titanic_data.drop(['name','ticket','cabin','boat','body','home.dest','sex','embarked','pclass'], axis=1)

# Set random seed for reproducibility.
np.random.seed(131254)

# Convert features and labels to numpy arrays.
labels = titanic_data["survived"].to_numpy()
titanic_data = titanic_data.drop(['survived'], axis=1)
feature_names = list(titanic_data.columns)
data = titanic_data.to_numpy()

# Separate training and test sets using 
train_indices = np.random.choice(len(labels), int(0.7*len(labels)), replace=False)
test_indices = list(set(range(len(labels))) - set(train_indices))
train_features = data[train_indices]
train_labels = labels[train_indices]
test_features = data[test_indices]
test_labels = labels[test_indices]

# MODEL

import torch
import torch.nn as nn
torch.manual_seed(1)  # Set seed for reproducibility.
class TitanicSimpleNNModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(12, 12)
        self.sigmoid1 = nn.Sigmoid()
        self.linear2 = nn.Linear(12, 8)
        self.sigmoid2 = nn.Sigmoid()
        self.linear3 = nn.Linear(8, 2)
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        lin1_out = self.linear1(x)
        sigmoid_out1 = self.sigmoid1(lin1_out)
        sigmoid_out2 = self.sigmoid2(self.linear2(sigmoid_out1))
        return self.softmax(self.linear3(sigmoid_out2))

net = TitanicSimpleNNModel()

# ENTRENAMIENTO

criterion = nn.CrossEntropyLoss()
num_epochs = 200

optimizer = torch.optim.Adam(net.parameters(), lr=0.1)
input_tensor = torch.from_numpy(train_features).type(torch.FloatTensor)
label_tensor = torch.from_numpy(train_labels)
for epoch in range(num_epochs):
  output = net(input_tensor)
  loss = criterion(output, label_tensor)
  optimizer.zero_grad()
  loss.backward()
  optimizer.step()
  if epoch % 20 ==0:
    print('Epoch {}/{} => Loss: {:.2f}'.format(epoch+1, num_epochs, loss.item()))

torch.save(net.state_dict(), 'models/titanic_model.pt')

So I send input_tensor in the init of TitanicSimpleNNNModel(nn.Module). But it is making a call with super to nn.Module which is a torch.nn. No idea what this thing is for.

Here is the log:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-39-ab215714ae42> in <module>()
     10 label_tensor = torch.from_numpy(train_labels)
     11 for epoch in range(num_epochs):
---> 12   output = net(input_tensor)
     13   loss = criterion(output, label_tensor)
     14   optimizer.zero_grad()

3 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1128         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130             return forward_call(*input, **kwargs)
   1131         # Do not call functions when jit is used
   1132         full_backward_hooks, non_full_backward_hooks = [], []

<ipython-input-34-9fde1a60e114> in forward(self, x)
     16     sigmoid_out1 = self.sigmoid1(lin1_out)
     17     sigmoid_out2 = self.sigmoid2(self.linear2(sigmoid_out1))
---> 18     return self.softmax(self.linear3(sigmoid_out2))

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1128         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130             return forward_call(*input, **kwargs)
   1131         # Do not call functions when jit is used
   1132         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py in forward(self, input)
    112 
    113     def forward(self, input: Tensor) -> Tensor:
--> 114         return F.linear(input, self.weight, self.bias)
    115 
    116     def extra_repr(self) -> str:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (916x2 and 12x8)

My first thought is that the data may not be up to date. But that would also surprise me.

Answer 1

The code you're using in the colab notebook is different than what you've posted here, mainly the definition for the second sigmoid layer. You are using a linear layer (nn.Linear) instead of a sigmoid layer. In addition, the number of neurons for your third linear layer is not correct, it should be eight instead of twelve since the second linear layer has eight neurons. All those changes have been incorporated into the model definition you've shared here, so using that code doesn't give any errors.