0
$\begingroup$

I am doing a time series data training. I have to pad 0s to the data so the sequences have the same length. Because of 0s are padded, I have to mask them during the training, for Keras, it is simply done by applying a Masking layer.

However, Pytorch requires much more steps. The pack_padded_sequence allows us to mask the 0s but the function requires me to place all the different length sequences in one list. I was stuck here because I always put all the aligned sequences to Dataloader and run the training. How can I run the batch training without the Dataloader.

A sample code would be the following

X = [[[1,2,3],[2,3,4]],[[1,2,4]],[[5,6,4]]] #  a list of 2d arrays.
y = [0,1,2]

model = model(ninp=40,num_layers=num_layers,class_num=class_num,nhid=nhid) 
#The desired input shape is (X,60,40) (batch,length,features)
loss_function = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.5e-3)

for x,y in zip(X_train,y_train):
   # x to be a small batch
   x_padded= pad_sequence(x, padding_value=0.0)
   lengths = torch.tensor([len(t) for t in x])
   x_packed= torch.nn.utils.rnn.pack_padded_sequence(x_padded, lengths.to('cpu'), enforce_sorted=False)
   optimizer.zero_grad()
   y_pred = model(x_packed)
   loss = loss_function(y_pred, y)    
   loss.backward()
   optimizer.step()

As it is shown that Dataloader is replaced with zip(). Am I on the right track?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.