I'm using the transformer architecture to predict future time-points from previous time-points.
Each item of the input sequence is a vector of [ temperature, time, sunlight ]
Each item of the output sequence is simply a temperature.

I want to also provide a contextual, non time-dependent feature input (e.g. altitude) to improve the prediction.

What's the best way to plug in that additional meta feature to the transformer? I've considered the following:

  1. Simply appending the constant feature to each item of the input sequence:
    [ temperature, time, sunlight, altitude ]

  2. Passing altitude through a couple of dense layers and using the output as a latent space representation to be inputted to one of the attention heads in every encoder layer. Basically replicating the decoder layers but with the altitude encoding as foreign input.

  3. Passing altitude through a couple of dense layers and using the output to balance the output of the encoder layers.

Is there an agreed upon best practice or available implementation for this use case?


1 Answer 1


The easiest approach is to plug-in the altitude for each input sequence. The other way, is to build an architecture where you have 2 set of inputs: one for the RNN and one for the constants and then add a concatenate level to combine the 2.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.