I am doing a time series data training. I have to pad 0s to the data so the sequences have the same length. Because of 0s are padded, I have to mask them during the training, for Keras, it is simply done by applying a Masking layer.

However, Pytorch requires much more steps. The pack_padded_sequence allows us to mask the 0s but the function requires me to place all the different length sequences in one list. I was stuck here because I always put all the aligned sequences to Dataloader and run the training. How can I run the batch training without the Dataloader.

A sample code would be the following

X = [[[1,2,3],[2,3,4]],[[1,2,4]],[[5,6,4]]] #  a list of 2d arrays.
y = [0,1,2]

model = model(ninp=40,num_layers=num_layers,class_num=class_num,nhid=nhid) 
#The desired input shape is (X,60,40) (batch,length,features)
loss_function = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.5e-3)

for x,y in zip(X_train,y_train):
   # x to be a small batch
   x_padded= pad_sequence(x, padding_value=0.0)
   lengths = torch.tensor([len(t) for t in x])
   x_packed= torch.nn.utils.rnn.pack_padded_sequence(x_padded, lengths.to('cpu'), enforce_sorted=False)
   y_pred = model(x_packed)
   loss = loss_function(y_pred, y)    

As it is shown that Dataloader is replaced with zip(). Am I on the right track?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.