What is the right Pytorch RNN implementation?

Question

I read about RNN in pytorch: RNN — PyTorch 1.12 documentation.

According to the document the RNN run the following function:

I looked on another RNN example (from pytorch tutorial): NLP FROM SCRATCH: CLASSIFYING NAMES WITH A CHARACTER-LEVEL RNN.

And they implemented RNN as:

import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

n_hidden = 128
rnn = RNN(n_letters, n_hidden, n_categories)

1. Why the implemented function is different from the equation ? (The function doesn’t contains softmax and it does contain bias which is not shown in the code)

2. Why the code dosn’t use tanh as shown in the equation ?

Answer 1

nn.Linear should already learn an additive bias by default.

This example implements RNN with no activation likely just for educational purposes, since that makes no sizeable difference from fully connected network (see the proof at https://stackoverflow.com/questions/66726974/pytorch-rnn-with-no-nonlinearity) and is questionable in other aspects as well (https://github.com/pytorch/tutorials/issues/193). tanh is usually preferred as an activation function for RNN connection since it's less prone to vanishing gradients problem than ReLU (which is a serious concern when running RNNs), allows for both increase and decrease of the hidden states and generally shows better convergence behaviour. The classification output can use any suitable activation.