# The use of Keras self-attention module

This question calls people to share their personal experiences with keras_self_attention module.

I also summarized the problems I encountered and the solutions I found or received from answers.

Background

I am building a classifier using time series data. The input is in shape of (batch, step, features).

The flawed codes are shown below.

import tensorflow as tf
from keras_self_attention import SeqSelfAttention

X_train = np.random.rand(700, 50,34)
y_train = np.random.choice([0, 1], 700)
X_test = np.random.rand(100, 50, 34)
y_test = np.random.choice([0, 1], 100)

model = tf.keras.models.Sequential()

model.compile(loss='categorical_crossentropy',
metrics=['accuracy'])

history = model.fit(X_train, y_train,
batch_size=batch_size,
epochs=epochs,
validation_data=(X_val, y_val),
verbose=1
)

yhat = model.predict_prob(X_test)

Problems

1. The model raised an IndexError at the SeqSelfAttention layer.

Solution: This problem was because the return_sequences in the last LSTM layer was not set to True

2. The model raised a ValueError before the Dense layer due to the shape incompatible problem.

Solution: The solution is to add a Flatten() layer before Dense layer.

*3. How do Muplicative attention and Multi-head work?

*4. Does stacking multiple layers of self_attention(SA) work for improving accuracy? Should it be like LSTM+LSTM+SA+SA or LSTM+SA+LSTM+SA?

• Can you specify what exact error message show? Feb 26, 2021 at 12:26
• @AIFahim I have posted the screenshot of the error
– Leo
Mar 1, 2021 at 6:20

It seems that the SeqSelfAttention layer is expecting all the time-steps.
i.e. return_sequences=True

import tensorflow as tf, numpy as np
from tensorflow import keras
from keras_self_attention import SeqSelfAttention

X_train = np.random.rand(700, 50,34)
y_train = np.random.choice([0, 1], 700)
X_test = np.random.rand(100, 50, 34)
y_test = np.random.choice([0, 1], 100)

model = tf.keras.models.Sequential()

model.summary()

model.compile(loss='binary_crossentropy')
model.fit(X_train,y_train,epochs=2)

• Thanks, that works as I somehow missed that point and I have updated the question as a ValueError occurred at the Dense layer. With referring to the model summary the output of the self_attention layer is still in time series. The homepage link used Dense without specifying the activation function so I tried that way but the same error popped out
– Leo
Mar 2, 2021 at 8:20
• Please add a Flatten layer before Dense. I am suggesting you only remedy of errors and cause. I have no idea what this module does. Just an FYI Mar 2, 2021 at 9:46
• Thank you for your help, you deserved the bounty and I will keep this question open and wait for other answers to share more experiences on the use of this module.
– Leo
Mar 9, 2021 at 7:26
• I will suggest you close this Question and Post a separate one with the precise questions adding a reference to this question. Mar 11, 2021 at 10:39