This question calls people to share their personal experiences with
I also summarized the problems I encountered and the solutions I found or received from answers.
I am building a classifier using time series data. The input is in shape of (batch, step, features).
The flawed codes are shown below.
import tensorflow as tf from tensorflow.keras.layers import Dense, Dropout,Bidirectional,Masking,LSTM from keras_self_attention import SeqSelfAttention X_train = np.random.rand(700, 50,34) y_train = np.random.choice([0, 1], 700) X_test = np.random.rand(100, 50, 34) y_test = np.random.choice([0, 1], 100) model = tf.keras.models.Sequential() model.add(Masking(mask_value=0.0, input_shape=(X_train.shape,X_train.shape))) model.add(Bidirectional(LSTM(units, dropout=dropout, recurrent_dropout=recurrent_dropout))) model.add(SeqSelfAttention(attention_activation='sigmoid')) model.add(Dense(num_classes, activation='softmax')) model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) history = model.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, validation_data=(X_val, y_val), verbose=1 ) yhat = model.predict_prob(X_test)
1. The model raised an IndexError at the
Solution: This problem was because the `return_sequences` in the last `LSTM` layer was not set to `True`
2. The model raised a
ValueError before the
Dense layer due to the shape incompatible problem.
Solution: The solution is to add a `Flatten()` layer before `Dense` layer.
*3. How do
Muplicative attention and
*4. Does stacking multiple layers of
self_attention(SA) work for improving accuracy? Should it be like LSTM+LSTM+SA+SA or LSTM+SA+LSTM+SA?