What exactly is the input of decoder in autoencoder setup

Question

I am reviewing various autoencoder setups for MNIST reconstruction, Seq2Seq translation and others. My naive understanding of data flow is as follows:

Input -> [Encoder] -> Hidden Representation -> [Decoder] -> Output.

However, in case of Seq2Seq translation task similar to Sutskever et. al. decoder input is combined from hidden state + input sequence.

I wonder how is the input of decoder dependent on the target task? Why do we need to put input sequence besides the hidden state? Any high-level explanation is appreciated.

Answer 1

Your are mixing two different beasts. Despite both having encoder and decoder parts, the way in which a normal feedforward image transformation network (i.e. the autoencoder) and an autoregressive model (i.e. the seq2seq) are actually used is very different:

• In the type of image transformation done in a vanilla autoencoder, you input an image and get another image at the output. That's it, both at training and at inference time.
• In an autoregressive model like seq2seq, you feed the network with the input and the initial tokens of the expected output sequence (i.e. the prefix), and you get the next token as output. During training, those initial tokens are normally from the gold data (aka teacher forcing), that is, you feed as prefix real tokens from the training data. In inference, those tokens are generated by the model itself: you generate the first token, then you generate the second token given the first one you previously generated, then you generate the third token given the previous 2 generated tokens, and so on.