How to use the keras.layers.AdditiveAttention correctly?

Question

My understanding on the topic is superficial at best, so do bear with me. I have a couple questions (specifically on how to use keras.layers.AdditiveAttention) which I hope is suitable to be asked together. There are also a number of similar questions posted, I do apologize if (that) I did not understand them properly to solve my problem.

1. For the attention mechanism, why must the dimensions of query and value be the same? E.g. Stacked 1a, and Stacked 3a.
2. It is my understanding that query:= last hidden state of the decoder, and values:= all hidden states of the encoder.
3. For my other examples (Stacked 1b and 2b) where there is no error, is the attention layer actually implemented correctly? If not, how should I do so?
4. In the case of the Single network, Stacked 3a and Stacked 3b, what should their respective query and value inputs be?

Context: I am trying to use multiple stacked of networks to extract features and then use the processed features for predictions.

I think most of these are repetitive, but I hope the examples help to illustrate my problem/confusion better.

##-------------------------------------------------------------------
## Some imports and functions
import numpy as np
import pandas as pd

from keras import Model
from keras.layers import Input, Dense, Dropout, RepeatVector, AdditiveAttention, GRU

def temporalize(X, y, past_records):
    """
    Taken from https://towardsdatascience.com/lstm-autoencoder-for-extreme-rare-event-classification-in-keras-ce209a224cfb      (I edited it by a tiny bit for my use case).
    """
    output_X = []
    output_y = []
    for i in range(len(X)-past_records):
        t = []
        t.append(X[i:(i+past_records+1)])
        output_X.append(t)
        output_y.append(y[i+past_records])
    return np.squeeze(np.array(output_X)), np.array(output_y)

##-------------------------------------------------------------------
## Random data
df = pd.concat([
    pd.Series(np.arange(30)), 
    pd.Series((np.arange(30))**2), 
    pd.Series((np.arange(30))**3),
    pd.Series((np.arange(30))**4)
    ], axis=1)
df.columns = ['A','B','Label_1','Label_2']

past_records = 4
n_batch      = 5

index_train = df.index[df.index[0:20]]
index_test  = df.index[df.index[20:]]

X_train          = df.drop(['Label_1','Label_2'], axis=1).loc[index_train]
y_train          = df[['Label_1','Label_2']].loc[index_train]
X_train, y_train = np.array(X_train), np.array(y_train)

X_test         = df.drop(['Label_1','Label_2'], axis=1).iloc[np.concatenate((index_train[-past_records:], index_test), axis=0)]
y_test         = df[['Label_1','Label_2']].iloc[np.concatenate((index_train[-past_records:], index_test), axis=0)]
X_test, y_test = np.array(X_test), np.array(y_test)

n_features = X_train.shape[1]
n_outputs  = y_train.shape[1]
X_train, y_train = temporalize(X=X_train, y=y_train, past_records=past_records)
X_train = X_train.reshape(X_train.shape[0], past_records+1, n_features)
X_test, y_test   = temporalize(X=X_test, y=y_test, past_records=past_records)
X_test = X_test.reshape(X_test.shape[0], past_records+1, n_features)

input_shape = (past_records+1, n_features)
inputs = Input(shape=input_shape, name='inputs')

##-------------------------------------------------------------------
## Single network
enc = GRU(8, return_sequences=True)(inputs)
# Attention
attn = AdditiveAttention()([enc, enc]) # query, value
out  = Dense(n_outputs, activation='relu')(attn)

# Compile
model = Model(inputs=inputs, outputs=out)
model.compile(optimizer='adam', loss='mse')
model.fit(X_train, np.array(y_train), epochs=3, batch_size=n_batch, shuffle=False)

##-------------------------------------------------------------------
## Stacked 1a (Error)
# InvalidArgumentError: Dimension must be equal, but are 4 and 8
enc = GRU(8, return_sequences=True)(inputs)
dec = GRU(4, return_sequences=False)(enc)

# Attention
attn = AdditiveAttention()([dec, enc]) # query, value
out  = Dense(n_outputs, activation='relu')(attn)
# Compile
model = Model(inputs=inputs, outputs=out)
model.compile(optimizer='adam', loss='mse')
model.fit(X_train, np.array(y_train), epochs=3, batch_size=n_batch, shuffle=False)

##-------------------------------------------------------------------
## Stacked 1b (In response to Stacked 1a)
enc = GRU(8, return_sequences=True)(inputs)
dec = GRU(8, return_sequences=False)(enc)

# Attention
attn = AdditiveAttention()([dec, enc]) # query, value
out  = Dense(n_outputs, activation='relu')(attn)
# Compile
model = Model(inputs=inputs, outputs=out)
model.compile(optimizer='adam', loss='mse')
model.fit(X_train, np.array(y_train), epochs=3, batch_size=n_batch, shuffle=False)


##-------------------------------------------------------------------
## Stacked 2a
enc = GRU(16, return_sequences=True)(inputs)
enc = GRU(8, return_sequences=True)(enc)
dec = GRU(8, return_sequences=False)(enc)

# Attention
attn = AdditiveAttention()([dec, enc]) # query, value
out  = Dense(n_outputs, activation='relu')(attn)
# Compile
model = Model(inputs=inputs, outputs=out)
model.compile(optimizer='adam', loss='mse')
model.fit(X_train, np.array(y_train), epochs=3, batch_size=n_batch, shuffle=False)

##-------------------------------------------------------------------
## Stacked 3a (Dimensions error again)
enc = GRU(16, return_sequences=True)(inputs)
enc = GRU(8, return_sequences=True)(enc)
dec = GRU(8, return_sequences=True)(enc)
dec = GRU(16, return_sequences=False)(dec )

# Attention
attn = AdditiveAttention()([dec, enc]) # query, value
out  = Dense(n_outputs, activation='relu')(attn)
# Compile
model = Model(inputs=inputs, outputs=out)
model.compile(optimizer='adam', loss='mse')
model.fit(X_train, np.array(y_train), epochs=3, batch_size=n_batch, shuffle=False)


##-------------------------------------------------------------------
## Stacked 3b
gru      = GRU(64, return_sequences=True)(inputs)
gru      = GRU(32, return_sequences=True)(gru)
gru      = GRU(16, return_sequences=True)(gru)
gru_last = GRU(8, return_sequences=False)(gru)

# Attention
attn = AdditiveAttention()([gru_last, ?]) # query, value
out  = Dense(n_outputs, activation='relu')(attn)
# Compile
model = Model(inputs=inputs, outputs=out)
model.compile(optimizer='adam', loss='mse')
model.fit(X_train, np.array(y_train), epochs=3, batch_size=n_batch, shuffle=False)

---

Some of the references/resources I have seen:

• https://stackoverflow.com/questions/56946995/how-to-build-a-attention-model-with-keras
• https://stats.stackexchange.com/questions/421935/what-exactly-are-keys-queries-and-values-in-attention-mechanisms
• How do Bahdanau - Luong Attentions use Query, Value, Key vectors?