# Neural network not learning at all

I am training a MLP on a tabular dataset, the pendigits dataset. Problem is that training loss and accuracy are more or less stable, while validation and test loss and accuracy are completely constant. The pendigits dataset contains 10 classes. My code is exactly the same with other experiments that I did for example on MNIST or CIFAR10 that work correctly. The only things that change are the dataset from MNIST/CIFAR10 to pendigits and the NN, from a ResNet-18 to a simple MLP. Below the training function and the network:

def train(net, loaders, optimizer, criterion, epochs=100, dev=dev, save_param = True, model_name="only-pendigits"):
torch.manual_seed(myseed)
try:
net = net.to(dev)
print(net)
# Initialize history
history_loss = {"train": [], "val": [], "test": []}
history_accuracy = {"train": [], "val": [], "test": []}
# Process each epoch
for epoch in range(epochs):
# Initialize epoch variables
sum_loss = {"train": 0, "val": 0, "test": 0}
sum_accuracy = {"train": 0, "val": 0, "test": 0}
# Process each split
for split in ["train", "val", "test"]:
# Process each batch
# Move to CUDA
input = input.to(dev)
labels = labels.to(dev)
# Compute output
pred = net(input)
#labels = labels.long()
loss = criterion(pred, labels)
# Update loss
sum_loss[split] += loss.item()
# Check parameter update
if split == "train":
loss.backward()
# Optimize
optimizer.step()
# Compute accuracy
_,pred_labels = pred.max(1)
batch_accuracy = (pred_labels == labels).sum().item()/input.size(0)
# Update accuracy
sum_accuracy[split] += batch_accuracy
scheduler.step()
# Compute epoch loss/accuracy
epoch_loss = {split: sum_loss[split]/len(loaders[split]) for split in ["train", "val", "test"]}
epoch_accuracy = {split: sum_accuracy[split]/len(loaders[split]) for split in ["train", "val", "test"]}
# Update history
for split in ["train", "val", "test"]:
history_loss[split].append(epoch_loss[split])
history_accuracy[split].append(epoch_accuracy[split])
# Print info
print(f"Epoch {epoch+1}:",
f"TrL={epoch_loss['train']:.4f},",
f"TrA={epoch_accuracy['train']:.4f},",
f"VL={epoch_loss['val']:.4f},",
f"VA={epoch_accuracy['val']:.4f},",
f"TeL={epoch_loss['test']:.4f},",
f"TeA={epoch_accuracy['test']:.4f},",
f"LR={optimizer.param_groups[0]['lr']:.5f},")
except KeyboardInterrupt:
print("Interrupted")
finally:
# Plot loss
plt.title("Loss")
for split in ["train", "val", "test"]:
plt.plot(history_loss[split], label=split)
plt.legend()
plt.show()
# Plot accuracy
plt.title("Accuracy")
for split in ["train", "val", "test"]:
plt.plot(history_accuracy[split], label=split)
plt.legend()
plt.show()


Network:

#TEXT NETWORK
class TextNN(nn.Module):

#Constructor
def __init__(self):
# Call parent contructor
super().__init__()
torch.manual_seed(myseed)
self.relu = nn.ReLU()
self.linear1 = nn.Linear(16, 128) #16 are the columns in input
self.linear2 = nn.Linear(128, 128)
self.linear3 = nn.Linear(128, 32)
self.linear4 = nn.Linear(32, 10)

def forward(self, tab):
tab = self.linear1(tab)
tab = self.relu(tab)
tab = self.linear2(tab)
tab = self.relu(tab)
tab = self.linear3(tab)
tab = self.relu(tab)
tab = self.linear4(tab)

return tab

model = TextNN()
print(model)


Is it possible that the model is too simple that it does not learn anything? I do not think so. I think that there is some error in training (but the function is exactly the same with the function I use for MNIST or CIFAR10 that works correctly), or in the data loading. Below is how I load the dataset:

pentrain = pd.read_csv("pendigits.tr.csv")

class TextDataset(Dataset):
"""Tabular and Image dataset."""

def __init__(self, excel_file, transform=None):
self.excel_file = excel_file
self.tabular = excel_file
self.transform = transform

def __len__(self):
return len(self.tabular)

def __getitem__(self, idx):
if torch.is_tensor(idx):
idx = idx.tolist()

tabular = self.tabular.iloc[idx, 0:]

y = tabular["class"]

tabular = tabular[['input1', 'input2', 'input3', 'input4', 'input5', 'input6', 'input7',
'input8', 'input9', 'input10', 'input11', 'input12', 'input13',
'input14', 'input15', 'input16']]
tabular = tabular.tolist()
tabular = torch.FloatTensor(tabular)

if self.transform:
tabular = self.transform(tabular)

return tabular, y

penditrain = TextDataset(excel_file=pentrain, transform=None)

train_size = int(0.80 * len(penditrain))
val_size = int((len(penditrain) - train_size))

pentrain, penval = random_split(penditrain, (train_size, val_size))

pentest = TextDataset(excel_file=pentest, transform=None)


All is loaded correctly, indeed if I print an example:

text_x, label_x = pentrain[0]
print(text_x.shape, label_x)
text_x

torch.Size([16]) 1
tensor([ 48.,  74.,  88.,  95., 100., 100.,  78.,  75.,  66.,  49.,  64.,  23.,
32.,   0.,   0.,   1.])


#Define generators
generator=torch.Generator()
generator.manual_seed(myseed)



I have been stuck with this problem for 2 days, and I do not know what the problem is...

EDIT: Basically, if I write print(list(net.parameters())) at the beginning of each epoch, I see that weights does never change, and for this reason loss and accuracy remain constant. Why weights are not changing??

From what I read, I see you are trying to perform a logistic regression by using linear and ReLU functions.

Since your model outputs the probability the input has to be in each class, I recommend to use function such as sigmoid or softmax in the last layer.

I post here the difference between using only linear and ReLU and using the sigmoid function.

Also in evaluation, the model with sigmoid has an accuracy of 98.8%.

New contributor
Iya Lee is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
• Hello. In PyTorch the CrossEntropyLoss already contains a softmax (It’s actually a LogSoftmax + NLLLoss combined into one function, pytorch.org/docs/stable/generated/…). Mar 16 at 14:37
• Hi, could you send me a full version of your code so i can copy and try it? maybe with a link Mar 16 at 14:46
• Sure, thank you: drive.google.com/file/d/1oeRdvtPdUKfbRNHGocov8X5fhZ6RwN9v/… Here you can find code and dataset Mar 16 at 14:52
• I tried to modify your code a couple of times and nothing changes, even with sigmoid. I'm sorry but i can't do other than to upvote question to increase its visibility Mar 16 at 15:29
• Thank you for the effort ;) Mar 16 at 15:34

I solved... mistake was that I was calling again model = TextNN() after instantiating the optimizer, so weights were not changing... So, every part was ok, apart from the optimizer that was working with another (unused) model.