# PyTorch CNN network outputs homogeneous results

I am a beginner at data science and I got a project where I want to do nlp via a convolutional neural network in PyTorch. The problem is that regardless of what comes out of the convolutional layers, the output from the perceptron is always repeated. It is as if it wasn't seeing anything.

I already tried to change the dataset, number of variables, loss function, optimization method and add or remove layers and nothing seems to happen. It's been a whole week that I'm working on this and I can't figure out what's wrong.

The loss function in this case is torch.nn.MultiLabelSoftMarginLoss, which is why I removed the Sigmoid activation on the very last layer. The optimization method is SGD. It converges within 15 runs or so.

Here's a picture of what the data looks like after it goes through the convolutional layer (I put the tensor layers side-by-side):

The height is the number of channels and the width is the number of features times the batch size.

Data for testing

Here are the y,X pairs I'm using for the training. I used torch.save to create the files.

Reproducible Example

import torch
from torch import nn

class model(nn.Module):
def __init__(self,Test,n_cat=1):
super(model, self).__init__()
self.last_kernel = 3
self.convolutions = nn.Sequential(
nn.ReLU(),
nn.MaxPool1d(kernel_size=3),
nn.ReLU(),
nn.MaxPool1d(kernel_size=3),
nn.ReLU(),
nn.ReLU(),
nn.ReLU(),
nn.MaxPool1d(kernel_size=self.last_kernel),
)
self.test_conv = self.convolutions(Test) #I know it's dumb but I just wanted to get it working
self.p_channels = self.test_conv.shape[1]*self.test_conv.shape[2]
self.perceptron = nn.Sequential(
nn.Linear(self.p_channels,100),
nn.Sigmoid(),
nn.Linear(100,16),
nn.Sigmoid(),
nn.Linear(16,n_cat),
#nn.Sigmoid()
)
def forward(self, x):
x = self.convolutions(x)
x = x.view(x.size(0), -1)
x = self.perceptron(x)
return x

m=model(T,y.shape[1])
loss_fn = torch.nn.MultiLabelSoftMarginLoss(reduction="sum")

r = 0.001 #learning rate

optimizer = torch.optim.SGD(m.parameters(), lr=r, momentum=0.9)

for t in range(15):
y_pred = m.forward(T)
loss = loss_fn(y_pred, y)
loss.backward()
optimizer.step()

#Final y:
y_final = torch.sigmoid(y_pred)



Output (first few)

In [1]: y_actual
Out[1]:
tensor([[0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
[0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
[0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
...,
[0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
[0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
[0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698]],
In [2]: y_actual.max(axis=1).indices
Out[2]:
tensor([24, 24, 24, 24, <keeps going>, 24, 24])


• might help if you added some test data and a training/test loop. Demonstrate how to reproduce your issue. – David Marx Oct 7 '19 at 15:41
• hope that's enough – Ícaro Lorran Oct 7 '19 at 16:15

Because of that, I simply made a transformation $$X\mapsto 2X-1$$ on the tensor, switching the zeros by -1's and the networks started giving heterogeneous results.