1
$\begingroup$

Hi I am new to RNN and have come across this the following implementation of Pytorchs LSTM, but I cant understand how (or why) the 'bias' and 'weight' strings work in the 'def init_weights'.

class LSTM_LM(nn.Module):
    
    def __init__(
            self,            
            pretrained_emb: torch.tensor,
            lstm_dim: int,       
            drop_prob: float = 0.0,
            lstm_layers: int = 1,
    ):
       

        
        super(LSTM_LM, self).__init__()

        
        self.vocab_size = pretrained_emb.shape[0]
        self.model = nn.ModuleDict({
            'embeddings': nn.Embedding.from_pretrained(pretrained_emb, padding_idx=pretrained_emb.shape[0] - 1),
            'lstm': nn.LSTM( 
                pretrained_emb.shape[1],
                lstm_dim,
                num_layers=lstm_layers,
                batch_first=True,
                dropout=dropout_prob),
            'ff': nn.Linear(lstm_dim, pre.shape[0]),
            'drop': nn.Dropout(dropout_prob)
        })

        # Initialize the weights of the model
        self._init_weights()

    def _init_weights(self):
        all_parameters = list(self.model['lstm'].named_parameters()) + \
                     list(self.model['ff'].named_parameters())
        for n, p in all_parameters:
            if 'weight' in n:
                nn.init.xavier_normal_(p)
            elif 'bias' in n:
                nn.init.zeros_(p)
 

EDIT To be more precise, what part of the code makes it possible to check if the string 'weight' appreas in n? n is as I understand it a parameter but does nn.LSTM consist of weight and bias as stringparameters such that I can access them with LSTM.parameter('weight')[1] for instance?

I am not sure how to understand it in relationhsip (if there is such) to the variable section of: https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html

Update I am now able to print all_parameters of LSTM. It looks like this:

[('weight_ih_l0', Parameter containing:
  tensor([[-0.5299,  0.0481],
          [-0.3032,  0.2907],
          [-0.0553, -0.4933],
          [-0.2063, -0.2334],
          [-0.5127, -0.1538],
          [-0.4484,  0.1707],
          [-0.3729,  0.3518],
          [-0.3200,  0.5846]], requires_grad=True)),
 ('weight_hh_l0', Parameter containing:
  tensor([[-0.6242,  0.5774],
          [ 0.7023, -0.3028],
          [-0.4403,  0.2972],
          [-0.3179,  0.4870],
          [ 0.2489,  0.0627],
          [ 0.6007,  0.3024],
          [-0.3393,  0.1481],
          [ 0.1212, -0.6172]], requires_grad=True)),
 ('bias_ih_l0', Parameter containing:
  tensor([-0.2282, -0.0345, -0.3226, -0.5983, -0.0105,  0.3180, -0.1699, -0.5312],
         requires_grad=True)),
 ('bias_hh_l0', Parameter containing:
  tensor([ 0.4270,  0.0965, -0.3981,  0.6470,  0.3207, -0.0163, -0.4651, -0.0321],
         requires_grad=True)),
 ('weight', Parameter containing:
  tensor([[ 0.2041,  0.5927],
          [ 0.4556,  0.1257],
          [ 0.5357, -0.1195],
          [ 0.0016, -0.1114]], requires_grad=True)),
 ('bias', Parameter containing:
  tensor([ 0.0932, -0.5147, -0.6265,  0.2009], requires_grad=True))]

Although I don't see how that match the variables in the pytorch documentation that I linked to above, such as:

~LSTM.weight_ih_l[k] – the learnable input-hidden weights of the \text{k}^{th}k th layer (W_ii|W_if|W_ig|W_io), of shape (4hidden_size, input_size) for k = 0. Otherwise, the shape is (4hidden_size, num_directions * hidden_size). If proj_size > 0 was specified, the shape will be (4*hidden_size, num_directions * proj_size) for k > 0

$\endgroup$
5
  • $\begingroup$ Perhaps the question isn't precise enough, should I elaborate w.r.t what I mean by 'how' and 'why' ? $\endgroup$
    – Piskator
    Commented Sep 2, 2022 at 9:28
  • $\begingroup$ It would indeed help to elaborate on what you mean with how/why. If you are looking for an explanation on what the code is doing then it's simply looping over all parameters and using a Xavier normal initialization for the weights and initializing the biases with a value of zero. $\endgroup$
    – Oxbowerce
    Commented Sep 2, 2022 at 10:28
  • $\begingroup$ @Oxbowerce, yes that part I agree on and understand the same way. I have tried to edit the post in order to elaborate my question. $\endgroup$
    – Piskator
    Commented Sep 2, 2022 at 11:58
  • $\begingroup$ The values you see in all_parameters match what is mentioned in the pytorch documentation under the 'Variables' header. For example, weight_ih_l0 and weight_hh_l0 in your code link back to the variables weight_ih_l[k] and weight_hh_l[k] that are mentioned in the documentation (with k being zero in this case). $\endgroup$
    – Oxbowerce
    Commented Sep 2, 2022 at 13:40
  • $\begingroup$ I think I understand. If you feel like putting it as a real answer, I shall approve it, and close the question. $\endgroup$
    – Piskator
    Commented Sep 2, 2022 at 18:30

1 Answer 1

0
$\begingroup$

The function _init_weights is simply looping over all parameters and using a Xavier normal initialization for the weights and initializing the biases with a value of zero. The values you see in all_parameters match what is mentioned in the pytorch documentation under the 'Variables' header. For example, weight_ih_l0 and weight_hh_l0 in your code link back to the variables weight_ih_l[k] and weight_hh_l[k] that are mentioned in the documentation (with k being zero in this case).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.