In the latest TensorFlow 2.1, the tensorflow.keras.layers submodule contains AdditiveAttention() and Attention() layers, implementing Bahdanau and Luong's attentions, respectively. (docs here and here.)

These new type of layers require query, value and key inputs (the latest is optional though). However, Query, Value, Key vectors are something I've always read referred to Transformer architectures.

What do these vectors represent, when it comes to Bahdanau and Luong attention? For example, if I want to train an RNN model for a common task (let's say time series forecast), what would these inputs represent?

EDIT: I'm thinking about a seq2seq to make forecasts. The input would be a series of given length, and a series of external variables. The output would be the series shifted forward of n steps.


The general formulation of attention with queries, keys and values corresponds to a re-retrieval view on attention: you have some queries that you use to retrieve some values based on keys that correspond to them.

With RNNs, attention is used for sequence-to-sequence models like machine translation. (Time series forecasting is usually formulated as sequence labeling.) The attention in the RNN decoder is a special case of this:

  • You only have one query which is the current RNN state. (Note that at training time you have access to all target words, so you can use the full set of queries.) In the original Bahdanau's paper, it is $s_{i-1}$ in Equation 6.

  • Keys and values are the same, they are the encoder states. In the Keras API, if you do not specify the keys, it uses the values as keys. In the Bahdanau's paper, it is $h_j$ in Equations 5 and 6.

An RNN decoder implemented in Keras then can look like this (based on the TensorFlow Tutorial):

class Decoder(tf.keras.Model):
  def __init__(self, vocab_size, embedding_dim, dec_units):
    super(Decoder, self).__init__()
    self.dec_units = dec_units
    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
    self.gru = tf.keras.layers.GRU(
        self.dec_units, return_sequences=True,
        return_state=True, recurrent_initializer='glorot_uniform')
    self.fc = tf.keras.layers.Dense(vocab_size)
    self.attention = tf.keras.layers.AdditiveAttention()

  def call(self, x, hidden, enc_output):
    # hidden is the previous hidden state (batch, 1, dec_units)
    # x is the previous output: (batch, 1)

    # enc_output shape == (batch_size, src_length, hidden_size)
    # hidden shape == (batch_size, 1, dec_units)
    context_vector = self.attention([hidden, enc_output])

    # x shape after passing through embedding == (batch_size, 1, embedding_dim)
    x = self.embedding(x)

    # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)
    x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)

    # passing the concatenated vector to the GRU
    output, state = self.gru(x)

    # output shape == (batch_size * 1, hidden_size)
    output = tf.reshape(output, (-1, output.shape[2]))

    # output shape == (batch_size, vocab)
    x = self.fc(output)

    return x, state
| improve this answer | |
  • $\begingroup$ I understand you explanation, thank you. It's not clear how is it compatible with the description and the examples provided in the TF docs I linked above. $\endgroup$ – Leevo Mar 3 at 13:21
  • $\begingroup$ The examples are strange. They have a bug, line 10 should have value_input It shows an architecture for sequence pair classification. They take two sequences, use one to retrieve something from the other and them average those pairs into a single vector. $\endgroup$ – Jindřich Mar 3 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.