5
$\begingroup$

This question calls people to share their personal experiences with keras_self_attention module.

I also summarized the problems I encountered and the solutions I found or received from answers.


Background

I am building a classifier using time series data. The input is in shape of (batch, step, features).

The flawed codes are shown below.

import tensorflow as tf
from tensorflow.keras.layers import Dense, Dropout,Bidirectional,Masking,LSTM
from keras_self_attention import SeqSelfAttention

X_train = np.random.rand(700, 50,34)
y_train = np.random.choice([0, 1], 700)
X_test = np.random.rand(100, 50, 34)
y_test = np.random.choice([0, 1], 100)

model = tf.keras.models.Sequential()
model.add(Masking(mask_value=0.0, input_shape=(X_train.shape[1],X_train.shape[2])))
model.add(Bidirectional(LSTM(units, dropout=dropout, recurrent_dropout=recurrent_dropout)))
model.add(SeqSelfAttention(attention_activation='sigmoid'))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss='categorical_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])

history = model.fit(X_train, y_train,
                        batch_size=batch_size,
                        epochs=epochs,
                        validation_data=(X_val, y_val),
                        verbose=1
                        )

yhat = model.predict_prob(X_test)

Problems

1. The model raised an IndexError at the SeqSelfAttention layer.

Solution: This problem was because the `return_sequences` in the last `LSTM` layer was not set to `True`

2. The model raised a ValueError before the Dense layer due to the shape incompatible problem.

Solution: The solution is to add a `Flatten()` layer before `Dense` layer.

*3. How do Muplicative attention and Multi-head work?

*4. Does stacking multiple layers of self_attention(SA) work for improving accuracy? Should it be like LSTM+LSTM+SA+SA or LSTM+SA+LSTM+SA?

$\endgroup$
2
  • $\begingroup$ Can you specify what exact error message show? $\endgroup$
    – AIFahim
    Feb 26 at 12:26
  • $\begingroup$ @AIFahim I have posted the screenshot of the error $\endgroup$
    – Leo
    Mar 1 at 6:20
5
+50
$\begingroup$

It seems that the SeqSelfAttention layer is expecting all the time-steps.
i.e. return_sequences=True
Same is shown in the home page example. Link

import tensorflow as tf, numpy as np
from tensorflow import keras 
from tensorflow.keras.layers import Dense, Dropout,Bidirectional,Masking,LSTM
from keras_self_attention import SeqSelfAttention

X_train = np.random.rand(700, 50,34)
y_train = np.random.choice([0, 1], 700)
X_test = np.random.rand(100, 50, 34)
y_test = np.random.choice([0, 1], 100)

model = tf.keras.models.Sequential()
model.add(keras.layers.Masking(mask_value=0.0, input_shape=(X_train.shape[1], X_train.shape[2])))
model.add(keras.layers.Bidirectional(LSTM(20, dropout=0.25, recurrent_dropout=0.1, return_sequences=True)))
model.add(SeqSelfAttention(attention_activation='sigmoid'))
model.add(keras.layers.Flatten())
model.add(Dense(1, activation='sigmoid'))

model.summary()

model.compile(loss='binary_crossentropy')
model.fit(X_train,y_train,epochs=2)

enter image description here

enter image description here

$\endgroup$
4
  • $\begingroup$ Thanks, that works as I somehow missed that point and I have updated the question as a ValueError occurred at the Dense layer. With referring to the model summary the output of the self_attention layer is still in time series. The homepage link used Dense without specifying the activation function so I tried that way but the same error popped out $\endgroup$
    – Leo
    Mar 2 at 8:20
  • 1
    $\begingroup$ Please add a Flatten layer before Dense. I am suggesting you only remedy of errors and cause. I have no idea what this module does. Just an FYI $\endgroup$
    – 10xAI
    Mar 2 at 9:46
  • $\begingroup$ Thank you for your help, you deserved the bounty and I will keep this question open and wait for other answers to share more experiences on the use of this module. $\endgroup$
    – Leo
    Mar 9 at 7:26
  • $\begingroup$ I will suggest you close this Question and Post a separate one with the precise questions adding a reference to this question. $\endgroup$
    – 10xAI
    Mar 11 at 10:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.