1
$\begingroup$

I have been working on modifying the transformer from the article The Annotated Transformer. One of the features I would like to include is the ability to pass a sequence of fixed length, and receive an output sequence of a shorter length, which is possible per this reference.

In my case, I am using a sequence of 10 randomly generated integers 0-9 for the input (just like the article) and trying to return a sequence of five 2s (this is the simplest attempt to get an output of a shorter length I could think of). The start of the sequence is denoted as 1, and the end of the sequence is not defined.

I am successfully able to send the encoder the "source" batch tensor, and the decoder the "target" batch tensor consisting of only 5 columns in the batch tensor. The transformer will train on this data, but it returns a sequence of length equal to the input.

What are the requirements of the transformer network to output a sequence of length that is not equal to the length of the input sequence?

Thanks in advance for any assistance

Improve this question
$\endgroup$
5
  • $\begingroup$ It is not clear what you are doing exactly. The Transformer has the inherent ability to generate variable-length sequences, you don't need to do anything special. The output of the Transformer decoder is always the same length as the input of the decoder. When using a Transformer at training time, we pass the whole target sequence (shifted one position to the right due to the addition of the "beginning of sequence" token at the beginning and the removal of the "end of sequence" token), so the output of the decoder has the same length as the target sequence. $\endgroup$
    – noe
    Commented Jan 22, 2023 at 8:25
  • $\begingroup$ When using a Transformer at inference time, we start by providing as input only the "beginning of sequence" token, and we obtain an output of length 1. We create a new input, attaching the previous output to the input of the previous iteration, and repeat the process. We do this until it generates the "end of sequence" token. In all cases the output of the decoder is the same length as the input of the decoder. $\endgroup$
    – noe
    Commented Jan 22, 2023 at 8:26
  • $\begingroup$ Please provide more details of what you are doing exactly. $\endgroup$
    – noe
    Commented Jan 22, 2023 at 8:27
  • $\begingroup$ Thank you for the responses, noe. I am only trying to generate an output whose length matches the target sequence, which is shorter than the source length. From what I can gather from your responses, the answer requires an EOS token? $\endgroup$
    – try_hard
    Commented Jan 22, 2023 at 16:39
  • $\begingroup$ Yes, you need EOS tokens to mark the end of the sequence. I have added an answer with all these pieces of information. $\endgroup$
    – noe
    Commented Jan 22, 2023 at 17:13

1 Answer 1

Reset to default
3
$\begingroup$

The Transformer has the inherent ability to generate variable-length sequences, you don't need to do anything special. The output of the Transformer decoder is always the same length as the input of the decoder.

When using a Transformer at training time, we pass the whole target sequence (shifted one position to the right due to the addition of the "beginning of sequence" token at the beginning and the removal of the "end of sequence" token), so the output of the decoder has the same length as the target sequence.

When using a Transformer at inference time, we start by providing as input only the "beginning of sequence" token, and we obtain an output of length 1. We create a new input, attaching the previous output to the input of the previous iteration, and repeat the process. We do this until it generates the "end of sequence" token. In all cases, the output of the decoder is the same length as the input of the decoder.

Note that you need EOS tokens to mark the end of the sequence, both for the source and target sequences. Both during training and inference. When preparing the input data for training, you remove the EOS token from the input target sequence, and prepend a BOS token to it.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I am a bit confused about the last paragraph. You say that BOS and EOS tokens are required for the target sequence in training, but then you say that the EOS token is removed from the target sequence for training and only use BOS. But that is what I have done and I am getting an output sequence from the decoder the same length as the input of the encoder instead of the desired length which is less than the length of the input to the encoder. $\endgroup$
    – try_hard
    Commented Jan 23, 2023 at 5:25
  • $\begingroup$ When you say "I am getting an output sequence from the decoder the same length as the input of the encoder" I understand that you are speaking about inference and that you have been decoding autoregressively token by token. Is that right? $\endgroup$
    – noe
    Commented Jan 23, 2023 at 7:23
  • $\begingroup$ that is correct, I have been training with a length 10 source sequence to the encoder and a length 5 target sequence to the decoder, but still get a length 10 sequence from the decoder with inference. $\endgroup$
    – try_hard
    Commented Jan 23, 2023 at 19:23
  • $\begingroup$ And how are the training and validation loss curves? $\endgroup$
    – noe
    Commented Jan 23, 2023 at 19:27
  • $\begingroup$ They begin around 3 and converge to about 0. It is the simplest attempt I could think of. $\endgroup$
    – try_hard
    Commented Jan 23, 2023 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.