RNN with PyTorch - I don't understand the initial parameters

Question

I would like to understand the pyTorch RNN module in detail. There I created a very simple and basic example:

import torch.nn as nn

# example input data
i_data = torch.arange(1,10).reshape((9,1))

rnn = nn.RNN(1, 1, 1, batch_first=False) # input_size, hidden_size, num_layers

# Predefined weight and bias values to recalculate the RNN's result. 
state_dict = rnn.state_dict()
state_dict["weight_ih_l0"] = torch.tensor([[2.0]])
state_dict["weight_hh_l0"] = torch.tensor([[3.0]])
state_dict["bias_ih_l0"]   = torch.tensor( [4.0] )
state_dict["bias_hh_l0"]   = torch.tensor( [5.0] )
rnn.load_state_dict(state_dict )

Now comes the incomprehensible for me ...

If I initialize the hidden_state with the following values

hidden_state    = torch.zeros(1,1, dtype=torch.float32)

and feed the input date to the model

y, hidden_state = rnn(i_data, hidden_state)

I got the error: assert (input.dim() in (2, 3)), f"RNN: Expected input to be 2-D or 3-D but received {input.dim()}-D tensor" AssertionError: RNN: Expected input to be 2-D or 3-D but received 1-D tensor

I tried also many different combinations like

• hidden_state = torch.zeros(1,9, dtype=torch.float32) -> raise RuntimeError(msg.format(expected_hidden_size, list(hx.size()))) RuntimeError: Expected hidden size (1, 1, 1), got [1, 1, 9]
• hidden_state = torch.zeros(1,1,9, dtype=torch.float32) -> raise RuntimeError( RuntimeError: For unbatched 2-D input, hx should also be 2-D but got 3-D tensor

I'm totally confused. Can someone please help me. Many thanks.

Answer 1

Running your code as is, I get an error about i_data being Long instead of Float. RuntimeError: expected scalar type Float but found Long. However, changing just the one line to

i_data = torch.arange(1,10).float().reshape((9,1))

and I get outputs y and hidden_state returned correctly.

hidden_state    = torch.zeros(1,1, dtype=torch.float32)
y, hidden_state = rnn(i_data, hidden_state)

The parameters to the constructor for RNN are input_size, hidden_size, num_layers, and the arguments you pass to rnn are input, hidden_0. Everything seems correct at first glance to me. Do you maybe have variables still initialized from previous runs or is this working with the one-line change for you?

Answer 2

Thank you for your answer, I solved the puzzle meanwhile, but it took me a while. The website from Akshaj Verma has helped me a lot.

The main problem was that I was confused with usage of batch_first, sequence length, batch size and the organisation of the tensors (data, h0 and the result).

First, I show you my code and then I explain it.

class DC_Network(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_size, sequence_length, batch_size):
        super(DC_Network, self).__init__()
        self.input_size       = input_size
        self.hidden_size      = hidden_size
        self.num_layers       = num_layers
        self.batch_size       = batch_size
        self.sequence_length  = sequence_length
        self.output_size      = output_size
        self.rnn = nn.RNN(input_size=self.input_size, hidden_size=self.hidden_size, num_layers = self.num_layers, batch_first=True, nonlinearity='relu')
        self.fc  = nn.Linear(self.hidden_size * self.sequence_length, self.output_size )


    def forward(self, x): 
        inputs  = x.unfold(dimension = 0,size = self.sequence_length, step = 1) # 2
        if list(inputs.shape)[-1] == 4:
            inputs = torch.swapaxes(inputs,1,2) 
        h0      =  torch.ones(self.num_layers , self.batch_size, self.hidden_size)
        out, _ = self.rnn(inputs, h0) 
        out = out.reshape(self.batch_size, self.hidden_size * self.sequence_length) # 5 
        out = self.fc(out)
        return out

    def train_model(self, train_data, train_labels, num_epochs, learning_rate=0.01):
        criterion = nn.MSELoss()
        optimizer = optim.Adam(self.parameters(), lr=learning_rate)   
        
        for epoch in range(num_epochs):
            total_loss = 0
            optimizer.zero_grad()
            output = self(train_data)
            loss = criterion(output, train_labels)
            total_loss += loss.item()
            
            loss.backward(retain_graph=True) 
            optimizer.step()

            if (epoch + 1) % 10 == 0:
                print(f"Epoch [{epoch + 1}/{num_epochs}], Loss: {total_loss}")

Understanding sequence and batch_size

For a RNN a sequence of the input values needs to be defined. The sequence can be an ordered list or different timestamps of the input data. A complete sequence is required to generate an output. The sequence length depends on the system. In PyTorch (and I think also in other neural network frameworks) a sequence for RNN is equivalent to one batch. (A batch is a subset of the data). The batch size defines how many batches are available in the input data or with other words in how many batches the input data can be divided (-which means for RNN in how many sequences the input data can be divided.

              [a1, a2, a3], # 1. sequence of input a = 1. batch
              [a4, a5, a6], # 3. sequence of input a = 2. batch
              [a7, a8, a9], # 3. sequence of input a = 3. batch
             ]
batch_size = len(batches) # 3

The RNN creates for every element in a sequence 1 neuron (because every element needs to be stored with its previous value. In addition, the hidden_size can be independently defined, which multiplies every neuron. Therefore, the output size of the RNN is sequence_length * hidden_size and that's also the reason why a linear layer (fully connected, fc) is needed to reduce the system output to the actual size (output_size).

Restructuring of the input_data

In the next step, it is necessary to reorganise the input_data into batches/sequences. In the following example I have two inputs a and b. These inputs are organized as 2-dimensional tensor x.

x = [[ a1, b1], 
     [ a2, b2],
     [ a3, b3],
     [ a4, b4],
     [ a5, b5],
     [ a6, b6],
     [ a7, b7],
     [ a8, b8],
     [ a9, b9],
     ...
    ]

This 2-dimensional tensor need to be reorganized in batches or sequences. This separations in batches creates an additional dimension to the tensor x, which brings us to the nn.RNN parameter batch_first=true. The batch or sequence must be identified by the first index of the reorganized 3-dimensional input data. The reorganization can be done with view or reshape (in pyTorch) or with unfold to create overlapping sequences.

inputs  = x.view(self.batch_size, self.sequence_length, self.input_size)
inputs = [
          [[a1, b1], 
           [a2, b2],
           [a3, b3]
          ], 
          [[a4, b4], 
           [a5, b5],
           [a6, b6]
          ], 
          [[a7, b7], 
           [a8, b8],
           [a9, b9]
          ]
         ]

inputs  = x.unfold(dimension = 0,size = self.sequence_length, step = 1)
inputs = [
          [[a1, b1], 
           [a2, b2],
           [a3, b3]
          ], 
          [[a2, b2], 
           [a3, b3],
           [a4, b4]
          ], 
          [[a3, b3], 
           [a4, b4],
           [a5, b5]
          ],
          ...
         ]

If you use unfold with more than one input it is necessary to reorganize the shape of the inputs, because row and column of the sequence are swapped.

inputs = torch.swapaxes(inputs,1,2) 

Initial hidden state

The hidden state of a RNN stores the result of the last calculation. Therefore, for the first calculation an initial value is needed. What I didn't know is that in PyTorch a hidden state have to be created for every batch. This confused me completely, because normally (or my understanding of RNN) I have only one initial state for all batches. Hence, I initialized all hidden states for all batches equally. In addition the hidden state is also needed for every layer.

h0 =  torch.ones(self.num_layers , self.batch_size, self.hidden_size)

Calculation of the output

The the correct organized inputs and hidden states h0 the calculation of the outputs can be easily done.

out, _ = self.rnn(inputs, h0)

The _ is the placeholder for the new hidden_state_values which I don't use in my case. For every epoch I initialize the hidden state new. However, as Akshaj Verma mentioned on his side the out tensor's organisation is independent to the parameter batch_first, see issue 4145. Therefore, the out shape has to be reorganized bevor it can be used in the linear layer.

out = out.reshape(self.batch_size, self.hidden_size * self.sequence_length)

Now, with this implementation I was able to learn successfully the dynamic behavior of a DC motor.