In the latest TensorFlow 2.1, the tensorflow.keras.layers submodule contains AdditiveAttention() and Attention() layers, implementing Bahdanau and Luong's attentions, respectively. (docs here and here.)

These new type of layers require query, value and key inputs (the latest is optional though). However, Query, Value, Key vectors are something I've always read referred to Transformer architectures.

What do these vectors represent, when it comes to Bahdanau and Luong attention? For example, if I want to train an RNN model for a common task (let's say time series forecast), what would these inputs represent?

EDIT: I'm thinking about a seq2seq to make forecasts. The input would be a series of given length, and a series of external variables. The output would be the series shifted forward of n steps.


2 Answers 2


To answer your specific questions:

  1. AdditiveAttention() and Attention() layers, are (loosely but not exactly) based on Bahdanau and Luong's attentions, respectively.

  2. They use post-2018 semantics of queries, values and keys. To map the semantics to the Bahdanau or Luong's paper, you can consider the 'query' to be the last decoder hidden state. The 'values' will be the set of the encoder outputs - all the hidden states of the encoder. The 'query' 'attends' to all the 'values'

  3. If you run thru' the library code you will see that the query is first expanded over the time axis and then there is a dense layer that determines the weights of w1, w2. These weights are applied to the expanded query and the values and then they are added and another weight 'v' is finally applied. A softmax of this is taken to return the attention weights and then these attention weights are multiplied with the 'values' and added to return the context. This is Bahdanau's additive logic

  4. However, while analysing tf.keras.layers.Attention Github code to better understand how to use the same, the first line I could come across was - "This class is suitable for Dense or CNN networks, and not for RNN networks". Since you are using RNN, I would use this layer with caution. In general, all these ready-to-use layers are mostly for self-attention and if you want to create a transformer like model where you are doing away with the RNN altogether and want too use only attention to represent the sequence, you can consider these classes.

  5. If you still want to use the same, you can go ahead and try the following:

    ##Input 1 = the last decoder hidden state: stminus1
    ##Input 2 = All hidden states of the encoder: lstm_out
    ##Apply Bahdanau additive attention and give me the 
    ##output = context
    context = tf.keras.layers.AdditiveAttention()([stminus1, lstm_out])

You can now use the context additionally to strengthen the predictions.

However I would strongly recommend you write your own attention layer in less than half a dozen lines of code. See for e.g.: https://stackoverflow.com/questions/63060083/create-an-lstm-layer-with-attention-in-keras-for-multi-label-text-classification/64853996#64853996

  • 1
    $\begingroup$ Thank you for this answer, it's very informative. Could you elaborate on the last point? Why do you recommend to write a custom attention layer, rather than using the official one? And why it is "suitable for Dense or CNN networks, and not for RNN networks"? Thank you $\endgroup$
    – Leevo
    Nov 28, 2020 at 21:49
  • 1
    $\begingroup$ The attention layer provided by Keras is tf.keras.layers.AdditiveAttention. If you open the source code for this, you will see the disclaimer quoted above. I am not sure why It should not be used for RNNs. I intend to raise a ticket to find out why. The only alternative left is to write a custom attention layer. Luckily the implementation is straightforward and can be done in less than a dozen lines of code. $\endgroup$
    – Allohvk
    Nov 29, 2020 at 6:59
  • $\begingroup$ A ticket about this would be super useful. Please keep us updated I'm super interested! $\endgroup$
    – Leevo
    Nov 29, 2020 at 11:03
  • $\begingroup$ Looks like there is already a ticket: github.com/keras-team/keras/issues/14246. It has been a month since someone raised it. No response so far :) $\endgroup$
    – Allohvk
    Nov 29, 2020 at 17:05

The general formulation of attention with queries, keys and values corresponds to a re-retrieval view on attention: you have some queries that you use to retrieve some values based on keys that correspond to them.

With RNNs, attention is used for sequence-to-sequence models like machine translation. (Time series forecasting is usually formulated as sequence labeling.) The attention in the RNN decoder is a special case of this:

  • You only have one query which is the current RNN state. (Note that at training time you have access to all target words, so you can use the full set of queries.) In the original Bahdanau's paper, it is $s_{i-1}$ in Equation 6.

  • Keys and values are the same, they are the encoder states. In the Keras API, if you do not specify the keys, it uses the values as keys. In the Bahdanau's paper, it is $h_j$ in Equations 5 and 6.

An RNN decoder implemented in Keras then can look like this (based on the TensorFlow Tutorial):

class Decoder(tf.keras.Model):
  def __init__(self, vocab_size, embedding_dim, dec_units):
    super(Decoder, self).__init__()
    self.dec_units = dec_units
    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
    self.gru = tf.keras.layers.GRU(
        self.dec_units, return_sequences=True,
        return_state=True, recurrent_initializer='glorot_uniform')
    self.fc = tf.keras.layers.Dense(vocab_size)
    self.attention = tf.keras.layers.AdditiveAttention()

  def call(self, x, hidden, enc_output):
    # hidden is the previous hidden state (batch, 1, dec_units)
    # x is the previous output: (batch, 1)

    # enc_output shape == (batch_size, src_length, hidden_size)
    # hidden shape == (batch_size, 1, dec_units)
    context_vector = self.attention([hidden, enc_output])

    # x shape after passing through embedding == (batch_size, 1, embedding_dim)
    x = self.embedding(x)

    # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)
    x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)

    # passing the concatenated vector to the GRU
    output, state = self.gru(x)

    # output shape == (batch_size * 1, hidden_size)
    output = tf.reshape(output, (-1, output.shape[2]))

    # output shape == (batch_size, vocab)
    x = self.fc(output)

    return x, state
  • $\begingroup$ I understand you explanation, thank you. It's not clear how is it compatible with the description and the examples provided in the TF docs I linked above. $\endgroup$
    – Leevo
    Mar 3, 2020 at 13:21
  • $\begingroup$ The examples are strange. They have a bug, line 10 should have value_input It shows an architecture for sequence pair classification. They take two sequences, use one to retrieve something from the other and them average those pairs into a single vector. $\endgroup$
    – Jindřich
    Mar 3, 2020 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.