The encoder of a seq2seq model is meant to generate a conditioning context for the decoder, as mentioned here
A RNN layer (or stack thereof) acts as "encoder": it processes the input sequence and returns its own internal state. Note that we discard the outputs of the encoder RNN, only recovering the state. This state will serve as the "context", or "conditioning", of the decoder in the next step.
Following the usual terminology and the implementation in the provided link, the encoder context is denoted as [c,h], where,
- c is the internal state of the encoder in the last time step
- h is the hidden state (output) of the encoder in the last time step
If one would like to add some external context e, what would be a reasonable way to do so?
- Generating new context using some NN architecture with inputs [c,h,e]?
- Concatenate the encoder context and the external context [(c,e),(h,e)]?
- Any other suggestion?
To clarify the problem, let's take one would like to predict the future trajectory of an object based on past positions and using the location of obstacles as external context