0
$\begingroup$

I am developing a CNN-LSTM autoencoder in pytorch to predict time sequences.

The CNN input is a RGB image:

RGB image => tensor[Batch size= 4, channel = 3,width= 256, height=256]

and the output is a tensor

tensor => tensor[batch_size=4, parameters= 4]

For example , 1 image have 4 types of time (initial, middle, transition, final). The CNN output is the times (tensor 4,4) and then this CNN output is the input in the LSTM.

The LSTM decoder have 2 inputs , the output cnn and caption (original times tensor[batch_size=4,parameters=4]

My decoder code is:

class DecoderRNN(nn.Module):
  def __init__(self,num_features,num_hidden,num_layers):
    super().__init__()
    
    self.hidden_size = num_hidden
    self.num_layers = num_layers
    self.lstm = nn.LSTM(input_size=num_features,hidden_size=hidden_size,num_layers=num_layers,batch_first=True)
    self.linear = nn.Linear(hidden_size,num_features)

  
  def init_hidden(self,batch_size):
    """ At the start of training, we need to initialize a hidden state;
    there will be none because the hidden state is formed based on previously seen data.
    So, this function defines a hidden state with all zeroes
    The axes semantics are (num_layers, batch_size, hidden_dim)
    """
    return (torch.zeros((self.num_layers, batch_size, self.hidden_size), device=device), \
            torch.zeros((self.num_layers, batch_size, self.hidden_size), device=device))
    
  
  def forward(self,coder,times):
    #print("coder:", coder.shape)
    #print("sequence:",times.shape)
    self.batch_size = coder.shape[0]
    self.hidden = self.init_hidden(self.batch_size)
    #h0,c0 = self.hidden
    #print("estado 1:",h0.shape)

    #print("estado 2:",c0.shape)

    embeddings = torch.cat((coder, times), dim=1)
    #print("vector latente:", embeddings.shape)
    lstm_out, self.hidden = self.lstm(embeddings.unsqueeze(2), self.hidden) # lstm_out shape : (batch_size, caption length, hidden_size) 
    outputs = self.linear(lstm_out) # outputs shape : (batch_size, caption length, vocab_size)
    return outputs

How could the decoder perform?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.