I want to use sequence-to-sequence architecture to generate sequences.

My data has such structure

[0, 0, 1, 0, ..., 0, 1] --> [12.34, 0.78, 1.54, 6.90, ..., 5.32]

I follow this tutorial to achieve it.

After forwarding through Encoder network encoder_hidden is used as a decoder_hidden. But what should I use as a first decoder_input to Decoder network?

The original tutorial uses a Start Of the Sequence token, but I can't use it because it is encoded as 0. Probably 0 as a number will give some additional information for decoder.


2 Answers 2


As you have seen, normally you need a "special token" to be given to the decoder as the first element in its input to start the autoregressive generation.

However, given that your output are real (floating point) numbers, it is a bit trickier, as you are not dealing with a discrete token vocabulary where you could simply reserve a token for that.

I would suggest using a specific value, like $0.0$. Your model should be able to figure out the no influence of the $0.0$ in the first position.

Another option would be to learn the value to be used as first token. You would have an extra trainable parameter that you use as value for the first position.

  • $\begingroup$ do you have maybe links where I can find more information about learning first token? $\endgroup$ Mar 4, 2020 at 17:38
  • $\begingroup$ I have never seen this been done before. However, it would only consist of having a new member variable declared in the class constructor, e.g. self.initial_value = nn.Parameter(torch.Tensor(1, 1)) and concatenating it to the input just like <SOS>. $\endgroup$
    – noe
    Mar 4, 2020 at 18:24

You can still use 0 as Start Of the Sequence token. Shift the input data by adding a constant to all values, for example adding 10. Then prepend a 0 to the input.

A linear transformation of input will not affect the ability of machine learning models to learn. Make sure to apply the same transformation to both training and prediction stages.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.