I have been trying to train a relatively simple two-tower net for recommendation. I am using PyTorch and the implementation is the following - basically embeddings layers for users and items, optional feed-forward net for both towers, dot product between the user and items representations, and sigmoid.

class SimpleTwoTower(nn.Module):
    def __init__(self, n_items, n_users, ln):
        super(SimpleTwoTower, self).__init__()
        self.ln = ln
        self.item_emb = nn.Embedding(num_embeddings=n_items, embedding_dim=self.ln[0])
        self.user_emb = nn.Embedding(num_embeddings=n_users, embedding_dim=self.ln[0])
        self.item_layers = [] #nn.ModuleList()
        self.user_layers = [] #nn.ModuleList()
        for i, n in enumerate(ln[0:-1]):
            m = int(ln[i+1])
            self.item_layers.append(nn.Linear(n, m, bias=True))
            self.user_layers.append(nn.Linear(n, m, bias=True))
        self.item_layers = nn.Sequential(*self.item_layers)
        self.user_layers = nn.Sequential(*self.user_layers)
        self.dot = torch.matmul
        self.sigmoid = nn.Sigmoid()
    def forward(self, items, users):
        item_emb = self.item_emb(items)
        user_emb = self.user_emb(users)
        item_emb = self.item_layers(item_emb)
        user_emb = self.user_layers(user_emb)

        dp = self.dot(user_emb, item_emb.t())
        return self.sigmoid(dp)

I am trining with Binary cross entropy loss and Adam optimizer. When I am using only the embeddings, I see improvements from epoch to epoch (loss is decreasing and the evaluation metric are increasing). However, once I add even a single feed-forward layer, the network learns just a bit in the first epoch and then stagnates. I have tried to had code one linear layer with ReLU, to check if the issue is with the way I am creating the list of layers, but this did not change anything.

Has anybody else had a similar problem?

Edit: Here I have posted the question in the PyTorch forum and I have some replies.

  • $\begingroup$ If not already, maybe worth asking at pytorch discuss forum as well. $\endgroup$
    – hH1sG0n3
    Apr 6, 2021 at 19:22
  • 1
    $\begingroup$ @hH1sG0n3 - good point, I will do this $\endgroup$ Apr 7, 2021 at 7:09
  • $\begingroup$ So it looks like it was a vanishing gradients problem in combination with batchnorm right? Feel free to write your own response below, this may be useful to track! $\endgroup$
    – hH1sG0n3
    Apr 8, 2021 at 12:21
  • $\begingroup$ Yeah, something like that - the gradients were getting to zero after about 3000 update step, but I am not sure if 'vanishing gradients' is the correct name for the problem. I will post a response with the solution for completeness. $\endgroup$ Apr 9, 2021 at 8:20
  • $\begingroup$ Fair, hadn't realised gradients were decreasing over iterations rather than layers. $\endgroup$
    – hH1sG0n3
    Apr 9, 2021 at 9:39

1 Answer 1


I now have a working network. It turned out that the gradients were all zeros after only about 3000 update step. I tried two approaches to fix this - using Batch Normalization after each activation function in the feed-forward net and changing the activation function from ReLU to Leaky ReLU. Both worked, and I ended up using the Leaky ReLU without normalization.

For the full thread in the PyTorch discussion forum here

  • $\begingroup$ Thanks for sharing, have you managed to identify why this is happening? $\endgroup$
    – hH1sG0n3
    Apr 9, 2021 at 9:40
  • 1
    $\begingroup$ One theory is that it is a "dead ReLU", but I am not 100% certain. $\endgroup$ Apr 12, 2021 at 12:37
  • 1
    $\begingroup$ So, I kind of tend to think that dead ReLus are linked to vanishing gradients. That is, if a weight matrix at layer L is a random rotation then inevitably half of the coordinates will always end up being negative, which ReLu will translate to zeros. Essentially at every step of the backprop linear operation the norm of that matrix is being slashed by a factor of 2 given a diagonal matrix of ReLu operation will zero out negative matrix elements. $\endgroup$
    – hH1sG0n3
    Apr 14, 2021 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.