Given standard transformer architecture with encoder and decoder.
What happens when the input for the encoder is shorter than the expected output from the decoder?
The decoder is expecting to receive value and key tensors from the encoder which size is dependent on the amount of input token.
I could solve this problem during training by padding input and outputs to the same size.
But how about inference, when I don't know the size of the output?
Should I make a prediction and if the decoder doesn't output the
stop token within range of available size, re-encode inputs with more padding and try again?
What are the common approaches to this problem? Thanks in advance, have a great day :)