I am developing a CNN-LSTM autoencoder in pytorch to predict time sequences.

The CNN input is a RGB image:

RGB image => tensor[Batch size= 4, channel = 3,width= 256, height=256]

and the output is a tensor

tensor => tensor[batch_size=4, parameters= 4]

For example , 1 image have 4 types of time (initial, middle, transition, final). The CNN output is the times (tensor 4,4) and then this CNN output is the input in the LSTM.

The LSTM decoder have 2 inputs , the output cnn and caption (original times tensor[batch_size=4,parameters=4]

My decoder code is:

class DecoderRNN(nn.Module):
  def __init__(self,num_features,num_hidden,num_layers):
    self.hidden_size = num_hidden
    self.num_layers = num_layers
    self.lstm = nn.LSTM(input_size=num_features,hidden_size=hidden_size,num_layers=num_layers,batch_first=True)
    self.linear = nn.Linear(hidden_size,num_features)

  def init_hidden(self,batch_size):
    """ At the start of training, we need to initialize a hidden state;
    there will be none because the hidden state is formed based on previously seen data.
    So, this function defines a hidden state with all zeroes
    The axes semantics are (num_layers, batch_size, hidden_dim)
    return (torch.zeros((self.num_layers, batch_size, self.hidden_size), device=device), \
            torch.zeros((self.num_layers, batch_size, self.hidden_size), device=device))
  def forward(self,coder,times):
    #print("coder:", coder.shape)
    self.batch_size = coder.shape[0]
    self.hidden = self.init_hidden(self.batch_size)
    #h0,c0 = self.hidden
    #print("estado 1:",h0.shape)

    #print("estado 2:",c0.shape)

    embeddings = torch.cat((coder, times), dim=1)
    #print("vector latente:", embeddings.shape)
    lstm_out, self.hidden = self.lstm(embeddings.unsqueeze(2), self.hidden) # lstm_out shape : (batch_size, caption length, hidden_size) 
    outputs = self.linear(lstm_out) # outputs shape : (batch_size, caption length, vocab_size)
    return outputs

How could the decoder perform?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.