The use of Keras self-attention module

Question

This question calls people to share their personal experiences with keras_self_attention module.

I also summarized the problems I encountered and the solutions I found or received from answers.

---

Background

I am building a classifier using time series data. The input is in shape of (batch, step, features).

The flawed codes are shown below.

import tensorflow as tf
from tensorflow.keras.layers import Dense, Dropout,Bidirectional,Masking,LSTM
from keras_self_attention import SeqSelfAttention

X_train = np.random.rand(700, 50,34)
y_train = np.random.choice([0, 1], 700)
X_test = np.random.rand(100, 50, 34)
y_test = np.random.choice([0, 1], 100)

model = tf.keras.models.Sequential()
model.add(Masking(mask_value=0.0, input_shape=(X_train.shape[1],X_train.shape[2])))
model.add(Bidirectional(LSTM(units, dropout=dropout, recurrent_dropout=recurrent_dropout)))
model.add(SeqSelfAttention(attention_activation='sigmoid'))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss='categorical_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])

history = model.fit(X_train, y_train,
                        batch_size=batch_size,
                        epochs=epochs,
                        validation_data=(X_val, y_val),
                        verbose=1
                        )

yhat = model.predict_prob(X_test)

Problems

1. The model raised an IndexError at the SeqSelfAttention layer.

Solution: This problem was because the `return_sequences` in the last `LSTM` layer was not set to `True`

2. The model raised a ValueError before the Dense layer due to the shape incompatible problem.

Solution: The solution is to add a `Flatten()` layer before `Dense` layer.

*3. How do Muplicative attention and Multi-head work?

*4. Does stacking multiple layers of self_attention(SA) work for improving accuracy? Should it be like LSTM+LSTM+SA+SA or LSTM+SA+LSTM+SA?

Answer 1

It seems that the SeqSelfAttention layer is expecting all the time-steps.
i.e. return_sequences=True
Same is shown in the home page example. Link

import tensorflow as tf, numpy as np
from tensorflow import keras 
from tensorflow.keras.layers import Dense, Dropout,Bidirectional,Masking,LSTM
from keras_self_attention import SeqSelfAttention

X_train = np.random.rand(700, 50,34)
y_train = np.random.choice([0, 1], 700)
X_test = np.random.rand(100, 50, 34)
y_test = np.random.choice([0, 1], 100)

model = tf.keras.models.Sequential()
model.add(keras.layers.Masking(mask_value=0.0, input_shape=(X_train.shape[1], X_train.shape[2])))
model.add(keras.layers.Bidirectional(LSTM(20, dropout=0.25, recurrent_dropout=0.1, return_sequences=True)))
model.add(SeqSelfAttention(attention_activation='sigmoid'))
model.add(keras.layers.Flatten())
model.add(Dense(1, activation='sigmoid'))

model.summary()

model.compile(loss='binary_crossentropy')
model.fit(X_train,y_train,epochs=2)