PyTorch CNN network outputs homogeneous results

Question

I am a beginner at data science and I got a project where I want to do nlp via a convolutional neural network in PyTorch. The problem is that regardless of what comes out of the convolutional layers, the output from the perceptron is always repeated. It is as if it wasn't seeing anything.

I already tried to change the dataset, number of variables, loss function, optimization method and add or remove layers and nothing seems to happen. It's been a whole week that I'm working on this and I can't figure out what's wrong.

The loss function in this case is torch.nn.MultiLabelSoftMarginLoss, which is why I removed the Sigmoid activation on the very last layer. The optimization method is SGD. It converges within 15 runs or so.

Here's a picture of what the data looks like after it goes through the convolutional layer (I put the tensor layers side-by-side):

The height is the number of channels and the width is the number of features times the batch size.

Data for testing

Here are the y,X pairs I'm using for the training. I used torch.save to create the files.

y: https://file.io/JiLnim

x: https://file.io/NKecI1

Reproducible Example

import torch
from torch import nn

T = torch.load("X_ex")
y = torch.load("y_ex")

class model(nn.Module):
    def __init__(self,Test,n_cat=1):
        super(model, self).__init__()
        self.last_kernel = 3
        self.convolutions = nn.Sequential(
            nn.Conv1d(Test.shape[1], 500, kernel_size=7, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=3),
            nn.Conv1d(500, 400, kernel_size=7, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=3),
            nn.Conv1d(400, 300, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv1d(300, 200, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv1d(200, 100, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=self.last_kernel),
            )
        self.test_conv = self.convolutions(Test) #I know it's dumb but I just wanted to get it working
        self.p_channels = self.test_conv.shape[1]*self.test_conv.shape[2]
        self.perceptron = nn.Sequential(
            nn.Linear(self.p_channels,100),
            nn.Sigmoid(),
            nn.Linear(100,16),
            nn.Sigmoid(),
            nn.Linear(16,n_cat),
            #nn.Sigmoid()
        )
    def forward(self, x):
        x = self.convolutions(x)
        x = x.view(x.size(0), -1)
        x = self.perceptron(x)
        return x

m=model(T,y.shape[1])
loss_fn = torch.nn.MultiLabelSoftMarginLoss(reduction="sum")

r = 0.001 #learning rate

optimizer = torch.optim.SGD(m.parameters(), lr=r, momentum=0.9)

for t in range(15):
    y_pred = m.forward(T)
    loss = loss_fn(y_pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

#Final y:
y_final = torch.sigmoid(y_pred)

Output (first few)

In [1]: y_actual
Out[1]: 
tensor([[0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        ...,
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698]],
       grad_fn=<SigmoidBackward>)
In [2]: y_actual.max(axis=1).indices
Out[2]: 
tensor([24, 24, 24, 24, <keeps going>, 24, 24])

Answer 1

So, I think I figured out what the problem was. It had nothing to do with the algorithm in itself but the data. The data which was being fed to the network was a sparse tensor; the thing is that the network wasn't being able to learn from its mistakes because every time it got it wrong it received a zero correction to the weight.

Because of that, I simply made a transformation $X\mapsto 2X-1$ on the tensor, switching the zeros by -1's and the networks started giving heterogeneous results.