I have a one layer lstm with pytorch on Mnist data. I know that for one layer lstm dropout option for lstm in pytorch does not operate. So, I have added a drop out at the beginning of second layer which is a fully connected layer. However, I observed that without dropout I get 97.75% accuracy on the test data and with dropout of 0.5 I get 95.36%. I want to ask whether I am doing something wrong or what is the reason for this phenomena? I change it into eval mode in test but I reach to 96.44% accuracy. Still it is less than without dropout. Thanks a lot

# RNN Model (Many-to-One)
class RNN(nn.Module):
   def __init__(self, input_size, hidden_size, num_layers, num_classes):
    super(RNN, self).__init__()
    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.lstm = nn.LSTM(input_size, hidden_size, num_layers,

    self.fc = nn.Sequential(
       nn.Linear(hidden_size*2, num_classes),
def init_hidden(self,x):

     return(Variable(torch.zeros(self.num_layers*2, x.size(0), self.hidden_size)).cuda(), 
     Variable(torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).cuda()))

def forward(self, x):
    # Set initial states 

    # Forward propagate RNN

    hidden = self.init_hidden(x)
    out, _ = self.lstm(x, hidden)  

    # Decode hidden state of last time step
    out = self.fc(out[:, -1, :])  
    return out

From what I have seen dropout for LSTMs should not be so high as 0.5 Recommendations are 0.1 or less.

This paper - Where to Apply Dropout in Recurrent Neural Networks for Handwriting Recognition has a detailed study on how to use Dropout with RNNs, and one of the recommendations is that 'it is almost always better before the lstm layer than inside or after it' (pg 3), so you can try that. However, I believe this depends on kind of data you have.

I remember reading a paper that had dropout for LSTM only being useful for a large LSTM (like 4096 unit x 4 layers). I cannot find it now, but in this one, the authors suggest something similar - showing dropout having better results on a 1500 unit x 2 layer lstm than a 650 unit by 2 layer, while the smaller network would just overfit. The lesson I guess depends on how big your lstm is and what you are trying to achieve - the use case being preventing overfitting while increasing model size.


It might be because of learning rate. You need to reduce your learning rate and let it run for more epochs.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.