I am a beginner at data science and I got a project where I want to do nlp via a convolutional neural network in PyTorch. The problem is that regardless of what comes out of the convolutional layers, the output from the perceptron is always repeated. It is as if it wasn't seeing anything.
I already tried to change the dataset, number of variables, loss function, optimization method and add or remove layers and nothing seems to happen. It's been a whole week that I'm working on this and I can't figure out what's wrong.
The loss function in this case is torch.nn.MultiLabelSoftMarginLoss
, which is why I removed the Sigmoid activation on the very last layer. The optimization method is SGD. It converges within 15 runs or so.
Here's a picture of what the data looks like after it goes through the convolutional layer (I put the tensor layers side-by-side):
The height is the number of channels and the width is the number of features times the batch size.
Data for testing
Here are the y,X pairs I'm using for the training. I used torch.save to create the files.
Reproducible Example
import torch
from torch import nn
T = torch.load("X_ex")
y = torch.load("y_ex")
class model(nn.Module):
def __init__(self,Test,n_cat=1):
super(model, self).__init__()
self.last_kernel = 3
self.convolutions = nn.Sequential(
nn.Conv1d(Test.shape[1], 500, kernel_size=7, stride=1, padding=1),
nn.ReLU(),
nn.MaxPool1d(kernel_size=3),
nn.Conv1d(500, 400, kernel_size=7, stride=1, padding=1),
nn.ReLU(),
nn.MaxPool1d(kernel_size=3),
nn.Conv1d(400, 300, kernel_size=3, stride=1, padding=1),
nn.ReLU(),
nn.Conv1d(300, 200, kernel_size=3, stride=1, padding=1),
nn.ReLU(),
nn.Conv1d(200, 100, kernel_size=3, stride=1, padding=1),
nn.ReLU(),
nn.MaxPool1d(kernel_size=self.last_kernel),
)
self.test_conv = self.convolutions(Test) #I know it's dumb but I just wanted to get it working
self.p_channels = self.test_conv.shape[1]*self.test_conv.shape[2]
self.perceptron = nn.Sequential(
nn.Linear(self.p_channels,100),
nn.Sigmoid(),
nn.Linear(100,16),
nn.Sigmoid(),
nn.Linear(16,n_cat),
#nn.Sigmoid()
)
def forward(self, x):
x = self.convolutions(x)
x = x.view(x.size(0), -1)
x = self.perceptron(x)
return x
m=model(T,y.shape[1])
loss_fn = torch.nn.MultiLabelSoftMarginLoss(reduction="sum")
r = 0.001 #learning rate
optimizer = torch.optim.SGD(m.parameters(), lr=r, momentum=0.9)
for t in range(15):
y_pred = m.forward(T)
loss = loss_fn(y_pred, y)
optimizer.zero_grad()
loss.backward()
optimizer.step()
#Final y:
y_final = torch.sigmoid(y_pred)
Output (first few)
In [1]: y_actual
Out[1]:
tensor([[0.0794, 0.0734, 0.0880, ..., 0.0817, 0.0864, 0.0698],
[0.0794, 0.0734, 0.0880, ..., 0.0817, 0.0864, 0.0698],
[0.0794, 0.0734, 0.0880, ..., 0.0817, 0.0864, 0.0698],
...,
[0.0794, 0.0734, 0.0880, ..., 0.0817, 0.0864, 0.0698],
[0.0794, 0.0734, 0.0880, ..., 0.0817, 0.0864, 0.0698],
[0.0794, 0.0734, 0.0880, ..., 0.0817, 0.0864, 0.0698]],
grad_fn=<SigmoidBackward>)
In [2]: y_actual.max(axis=1).indices
Out[2]:
tensor([24, 24, 24, 24, <keeps going>, 24, 24])