I am trying to build a sequence to sequence model. I have used Vision transformer as the encoder and LSTM with 1 layers as the decoder. The output of the encoder is given as the hidden state for decoder LSTM and it tries to predict the caption. Is this way of doing image captioning wrong? The model is not working, tried tuning all the hyperparaeters and the hidden state size also.