# pytorchs LSTMs use of 'bias' and 'weight' strings

Hi I am new to RNN and have come across this the following implementation of Pytorchs LSTM, but I cant understand how (or why) the 'bias' and 'weight' strings work in the 'def init_weights'.

class LSTM_LM(nn.Module):

def __init__(
self,
pretrained_emb: torch.tensor,
lstm_dim: int,
drop_prob: float = 0.0,
lstm_layers: int = 1,
):

super(LSTM_LM, self).__init__()

self.vocab_size = pretrained_emb.shape[0]
self.model = nn.ModuleDict({
'lstm': nn.LSTM(
pretrained_emb.shape[1],
lstm_dim,
num_layers=lstm_layers,
batch_first=True,
dropout=dropout_prob),
'ff': nn.Linear(lstm_dim, pre.shape[0]),
'drop': nn.Dropout(dropout_prob)
})

# Initialize the weights of the model
self._init_weights()

def _init_weights(self):
all_parameters = list(self.model['lstm'].named_parameters()) + \
list(self.model['ff'].named_parameters())
for n, p in all_parameters:
if 'weight' in n:
nn.init.xavier_normal_(p)
elif 'bias' in n:
nn.init.zeros_(p)



EDIT To be more precise, what part of the code makes it possible to check if the string 'weight' appreas in n? n is as I understand it a parameter but does nn.LSTM consist of weight and bias as stringparameters such that I can access them with LSTM.parameter('weight')[1] for instance?

I am not sure how to understand it in relationhsip (if there is such) to the variable section of: https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html

Update I am now able to print all_parameters of LSTM. It looks like this:

[('weight_ih_l0', Parameter containing:
tensor([[-0.5299,  0.0481],
[-0.3032,  0.2907],
[-0.0553, -0.4933],
[-0.2063, -0.2334],
[-0.5127, -0.1538],
[-0.4484,  0.1707],
[-0.3729,  0.3518],
('weight_hh_l0', Parameter containing:
tensor([[-0.6242,  0.5774],
[ 0.7023, -0.3028],
[-0.4403,  0.2972],
[-0.3179,  0.4870],
[ 0.2489,  0.0627],
[ 0.6007,  0.3024],
[-0.3393,  0.1481],
('bias_ih_l0', Parameter containing:
tensor([-0.2282, -0.0345, -0.3226, -0.5983, -0.0105,  0.3180, -0.1699, -0.5312],
('bias_hh_l0', Parameter containing:
tensor([ 0.4270,  0.0965, -0.3981,  0.6470,  0.3207, -0.0163, -0.4651, -0.0321],
('weight', Parameter containing:
tensor([[ 0.2041,  0.5927],
[ 0.4556,  0.1257],
[ 0.5357, -0.1195],
('bias', Parameter containing:
tensor([ 0.0932, -0.5147, -0.6265,  0.2009], requires_grad=True))]


Although I don't see how that match the variables in the pytorch documentation that I linked to above, such as:

~LSTM.weight_ih_l[k]  – the learnable input-hidden weights of the \text{k}^{th}k th layer (W_ii|W_if|W_ig|W_io), of shape (4hidden_size, input_size) for k = 0. Otherwise, the shape is (4hidden_size, num_directions * hidden_size). If proj_size > 0 was specified, the shape will be (4*hidden_size, num_directions * proj_size) for k > 0

• Perhaps the question isn't precise enough, should I elaborate w.r.t what I mean by 'how' and 'why' ? Sep 2, 2022 at 9:28
• It would indeed help to elaborate on what you mean with how/why. If you are looking for an explanation on what the code is doing then it's simply looping over all parameters and using a Xavier normal initialization for the weights and initializing the biases with a value of zero. Sep 2, 2022 at 10:28
• @Oxbowerce, yes that part I agree on and understand the same way. I have tried to edit the post in order to elaborate my question. Sep 2, 2022 at 11:58
• The values you see in all_parameters match what is mentioned in the pytorch documentation under the 'Variables' header. For example, weight_ih_l0 and weight_hh_l0 in your code link back to the variables weight_ih_l[k] and weight_hh_l[k] that are mentioned in the documentation (with k being zero in this case). Sep 2, 2022 at 13:40
• Ah, I see. Thank you. Sep 2, 2022 at 18:28

The function _init_weights is simply looping over all parameters and using a Xavier normal initialization for the weights and initializing the biases with a value of zero. The values you see in all_parameters match what is mentioned in the pytorch documentation under the 'Variables' header. For example, weight_ih_l0 and weight_hh_l0 in your code link back to the variables weight_ih_l[k] and weight_hh_l[k] that are mentioned in the documentation (with k being zero in this case).