pytorchs LSTMs use of 'bias' and 'weight' strings

Question

Hi I am new to RNN and have come across this the following implementation of Pytorchs LSTM, but I cant understand how (or why) the 'bias' and 'weight' strings work in the 'def init_weights'.

class LSTM_LM(nn.Module):
    
    def __init__(
            self,            
            pretrained_emb: torch.tensor,
            lstm_dim: int,       
            drop_prob: float = 0.0,
            lstm_layers: int = 1,
    ):
       

        
        super(LSTM_LM, self).__init__()

        
        self.vocab_size = pretrained_emb.shape[0]
        self.model = nn.ModuleDict({
            'embeddings': nn.Embedding.from_pretrained(pretrained_emb, padding_idx=pretrained_emb.shape[0] - 1),
            'lstm': nn.LSTM( 
                pretrained_emb.shape[1],
                lstm_dim,
                num_layers=lstm_layers,
                batch_first=True,
                dropout=dropout_prob),
            'ff': nn.Linear(lstm_dim, pre.shape[0]),
            'drop': nn.Dropout(dropout_prob)
        })

        # Initialize the weights of the model
        self._init_weights()

    def _init_weights(self):
        all_parameters = list(self.model['lstm'].named_parameters()) + \
                     list(self.model['ff'].named_parameters())
        for n, p in all_parameters:
            if 'weight' in n:
                nn.init.xavier_normal_(p)
            elif 'bias' in n:
                nn.init.zeros_(p)
 

EDIT To be more precise, what part of the code makes it possible to check if the string 'weight' appreas in n? n is as I understand it a parameter but does nn.LSTM consist of weight and bias as stringparameters such that I can access them with LSTM.parameter('weight')[1] for instance?

I am not sure how to understand it in relationhsip (if there is such) to the variable section of: https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html

Update I am now able to print all_parameters of LSTM. It looks like this:

[('weight_ih_l0', Parameter containing:
  tensor([[-0.5299,  0.0481],
          [-0.3032,  0.2907],
          [-0.0553, -0.4933],
          [-0.2063, -0.2334],
          [-0.5127, -0.1538],
          [-0.4484,  0.1707],
          [-0.3729,  0.3518],
          [-0.3200,  0.5846]], requires_grad=True)),
 ('weight_hh_l0', Parameter containing:
  tensor([[-0.6242,  0.5774],
          [ 0.7023, -0.3028],
          [-0.4403,  0.2972],
          [-0.3179,  0.4870],
          [ 0.2489,  0.0627],
          [ 0.6007,  0.3024],
          [-0.3393,  0.1481],
          [ 0.1212, -0.6172]], requires_grad=True)),
 ('bias_ih_l0', Parameter containing:
  tensor([-0.2282, -0.0345, -0.3226, -0.5983, -0.0105,  0.3180, -0.1699, -0.5312],
         requires_grad=True)),
 ('bias_hh_l0', Parameter containing:
  tensor([ 0.4270,  0.0965, -0.3981,  0.6470,  0.3207, -0.0163, -0.4651, -0.0321],
         requires_grad=True)),
 ('weight', Parameter containing:
  tensor([[ 0.2041,  0.5927],
          [ 0.4556,  0.1257],
          [ 0.5357, -0.1195],
          [ 0.0016, -0.1114]], requires_grad=True)),
 ('bias', Parameter containing:
  tensor([ 0.0932, -0.5147, -0.6265,  0.2009], requires_grad=True))]

Although I don't see how that match the variables in the pytorch documentation that I linked to above, such as:

~LSTM.weight_ih_l[k] – the learnable input-hidden weights of the \text{k}^{th}k th layer (W_ii|W_if|W_ig|W_io), of shape (4hidden_size, input_size) for k = 0. Otherwise, the shape is (4hidden_size, num_directions * hidden_size). If proj_size > 0 was specified, the shape will be (4*hidden_size, num_directions * proj_size) for k > 0

Answer 1

The function _init_weights is simply looping over all parameters and using a Xavier normal initialization for the weights and initializing the biases with a value of zero. The values you see in all_parameters match what is mentioned in the pytorch documentation under the 'Variables' header. For example, weight_ih_l0 and weight_hh_l0 in your code link back to the variables weight_ih_l[k] and weight_hh_l[k] that are mentioned in the documentation (with k being zero in this case).