0
$\begingroup$

I am doing a NLP sentiment analysis task using an LSTM model (which currently gives me a 50% test accuracy as compared to 84% of a Naive Bayes). It is a text corpus of movie reviews from here (https://ai.stanford.edu/~amaas/data/sentiment/)

I have the following data shapes and parameters.

sent_tok_train.shape
(1400, 500)
sent_tok_test.shape
(400, 500)
sent_tok_val.shape
(200, 500)


EMBEDDING_DIM = 50 (taken from the glove embedding file, glove.6B.50d.txt)
MAXLEN = 500 
VOCAB_SIZE =  33713

DENSE1_DIM = 64
DENSE2_DIM = 32

LSTM1_DIM = 32
LSTM2_DIM = 16

WD = 0.001

Now, I have two model versions.

# Model 1
model_lstm = tf.keras.Sequential([
   tf.keras.layers.Embedding(VOCAB_SIZE+1, EMBEDDING_DIM, input_length=MAXLEN,weights=[EMBEDDINGS_MATRIX], trainable=False),
   tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(LSTM1_DIM, dropout=0.5, kernel_regularizer = regularizers.l2(WD))),
   tf.keras.layers.Dense(1, activation='sigmoid')
])

# Set the training parameters
model_lstm.compile(loss='binary_crossentropy',
                  optimizer=tf.keras.optimizers.Adam(), 
                  metrics=[tf.keras.metrics.BinaryAccuracy()])

# Model 2 - stacked
model_lstm = tf.keras.Sequential([
   tf.keras.layers.Embedding(VOCAB_SIZE+1, EMBEDDING_DIM, input_length=MAXLEN,weights=[EMBEDDINGS_MATRIX], trainable=False),
   tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(LSTM1_DIM, dropout=0.5, kernel_regularizer = regularizers.l2(WD), return_sequences=True)), 
  # tf.keras.layers.LSTM(LSTM1_DIM,dropout = 0.5, return_sequences=True),
   tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(LSTM2_DIM, dropout=0.5, kernel_regularizer = regularizers.l2(WD))),
  # tf.keras.layers.LSTM(LSTM2_DIM),
  # tf.keras.layers.Dropout(0.5),
   tf.keras.layers.Dense(DENSE1_DIM, activation='relu'),#, kernel_regularizer = regularizers.l2(WD)), 
   tf.keras.layers.Dense(DENSE2_DIM, activation='relu'),
   tf.keras.layers.Dense(1, activation='sigmoid')
])

# Set the training parameters
model_lstm.compile(loss='binary_crossentropy',
                  optimizer=tf.keras.optimizers.Adam(), 
                  metrics=[tf.keras.metrics.BinaryAccuracy()])


Q1. which one would you think is a better one, model 1 or 2?

v1. # model 1 with 350 epochs

enter image description here Test loss: 1.341 Test binary accuracy: 50.50%

v2. # model 2 with 350 epochs when I fit the model, it gives slightly different results on accuracy and loss. enter image description here Test loss: 1.741 Test binary accuracy: 49.75%

Q2. what would be the best number of epochs? by these pictures, it looks like around 70? because after that, the two curves bifurcate.

When I run 70 epochs, I get

#Model 1
Test loss: 1.565
Test binary accuracy: 51.50%

#Model 2:
Test loss: 0.774
Test binary accuracy: 52.75%

Q3. Is it possible to increase the validation accuracy at all by hyperparameter tuning, or changing layers, or is an LSTM just a bad model for the task above?

Thank you very much. I would be greatful for any links as to learn more about this.

$\endgroup$
3
  • $\begingroup$ It seems that both of your models are overfitting to your training data. The first one has a stable validation loss up until epoch 70 after which is even increases, indicating that it is unable to generalize the patterns it is recognizing from the training data. The second one has an increasing loss from the start. I would therefore try to see if you can combat this by for example changing hyperparameters or using a model with fewer parameters. $\endgroup$
    – Oxbowerce
    Apr 6, 2022 at 13:50
  • $\begingroup$ well. I have tried to change maxlen (500, 1000, 1400), embedding (50, 100, 200), and different dimensions of my LSTM and Dense layers. yet no difference so far. what do you think about the number of epochs? $\endgroup$
    – Bluetail
    Apr 6, 2022 at 14:06
  • $\begingroup$ The model you are using has too many parameters but training data is to less..Please try a model with less parameters.. $\endgroup$ Apr 6, 2022 at 14:56

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.