I am doing a time series data training. I have to pad 0s to the data so the sequences have the same length. Because of 0s are padded, I have to mask them during the training, for Keras, it is simply done by applying a Masking layer.
However, Pytorch requires much more steps. The pack_padded_sequence allows us to mask the 0s but the function requires me to place all the different length sequences in one list. I was stuck here because I always put all the aligned sequences to Dataloader and run the training. How can I run the batch training without the Dataloader.
A sample code would be the following
X = [[[1,2,3],[2,3,4]],[[1,2,4]],[[5,6,4]]] # a list of 2d arrays.
y = [0,1,2]
model = model(ninp=40,num_layers=num_layers,class_num=class_num,nhid=nhid)
#The desired input shape is (X,60,40) (batch,length,features)
loss_function = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.5e-3)
for x,y in zip(X_train,y_train):
# x to be a small batch
x_padded= pad_sequence(x, padding_value=0.0)
lengths = torch.tensor([len(t) for t in x])
x_packed= torch.nn.utils.rnn.pack_padded_sequence(x_padded, lengths.to('cpu'), enforce_sorted=False)
optimizer.zero_grad()
y_pred = model(x_packed)
loss = loss_function(y_pred, y)
loss.backward()
optimizer.step()
As it is shown that Dataloader is replaced with zip(). Am I on the right track?
