Why does the classic Neural Network perform better than LSTM in Sentiment Analysis

Question

My goal is to predict the polarity of some reviews (negative, positive or neutral). I tried two different neural networks:

    left_branch = Input((7000, ))
    left_branch_dense = Dense(512, activation = 'relu')(left_branch)

    right_branch = Input((14012, ))
    right_branch_dense = Dense(512, activation = 'relu')(right_branch)
    merged = Concatenate()([left_branch_dense, right_branch_dense])
    output_layer = Dense(3, activation = 'softmax')(merged)

    model = Model(inputs=[left_branch, right_branch], outputs=output_layer)
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    model.fit([np.array(review_matrix), np.array(X_train)], labels,epochs=2, verbose=1)
    model.save('model.merged') 

    #############################################################################


    #############################################################################

    #We will try to merge two different models in a different way: Accuracy: 70

    # Prepare the review column for embedding: 
    review_matrix_for_embedding = prepare_for_encoding(train_set[4].tolist(), 7000) # Shape: (1503,100)        

    second_matrix = np.array(pd.concat([onehot_category, aspect_matrix],axis=1))


    left_branch = Input(shape=(100,), dtype='int32')
    # input_dim: Size of maximum integer (7001 here); output dim: Size of embedded vector; 
    # input_length: Size of the array
    left_branch_embedding = Embedding(7000, 300, input_length=100)(left_branch)
    lstm_out = LSTM(256)(left_branch_embedding)
    lstm_out = Dropout(0.7)(lstm_out)
    lstm_out = Dense(128, activation='sigmoid')(lstm_out)

    right_branch = Input((7012, ))
    merged = Concatenate()([lstm_out, right_branch])
    output_layer = Dense(3, activation = 'softmax')(merged)

    model = Model(inputs=[left_branch, right_branch], outputs=output_layer)
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    model.fit([review_matrix_for_embedding, second_matrix], labels,epochs=5, verbose=1)

The first one does 80% accuracy while the second one does 70%, with embedding vectors and LSTM layer. How is it possible? Is there anything wrong in my architecture?

Answer 1

First of all, I have noticed that you have used sigmoid activation function for your LSTM Dense Layer and in your ANN you used relu, maybe, MAYBE, this can be a reason for your lower performance. That is could be happening because sigmoid functions suffer from two problems: - Saturation of gradients: sigmoid functions have tail distributions, meaning that they saturate in this 'flat' regions practically diminishing the gradient to zero and affecting backpropagation/training process. - Sigmoid outputs are not zero-centered: This is an issue due the gradient calculation during backpropagation. Either having all enters positivo or negative will add a 'zigzag' effect difficulting the training process.

My comments above are taken from this excellent tutorial that you should read: http://cs231n.github.io/neural-networks-1/ I tried to summarize it but they did a master work and I think you should read it.

Second, we must consider other factors such as: are you analyzing your train/test/val losses? Maybe your LSTM networks just takes longer to train and reach its minimum. You need to work a little more on these parameters before taking any conclusions. Plot a graph showing your training and validation loss so we can assess if your model is underfitting.

Lastly, I have a question for you: Why should your LSTM network perform better than a simple ANN? Although LSTM + Embeddings are powerful techniques that have gained attention in a lot of fields, essentially NLP, that will be not every task that they beat classical approaches. I myself have tried with different data sets, and depending on the application, simple ML algorithms such as SVM would easily beat the more complex ones, including sentiment analysis.

So to conclude, try these things and let us know your results. Also, if anyone disagrees with my answer, I would like to discuss it. I hope it helps.

Answer 2

Thank you for your answer. I changed sigmoid for RELU, and the result is the same. Anyway, I will keep RELU now!!

Here are pictures of training for both the one who does 79% accuracy (merged two classic neural networks), and the one in which I do 70: (of course I don't talk about the training accuracy but the test accuracy). I see a difference in the loss value, but I don't know how to interpret it? Does this mean that for the less good architecture, I don't reach a minimum? If yes, what can I modify in my NN ?

Thank you for the time you took to answer!!

Edit: Where can I modify parameters like learning rate in the LSTM part of the Neural network?

    left_branch = Input(shape=(100,), dtype='int32')
    # input_dim: Size of maximum integer (7001 here); output dim: Size of embedded vector; 
    # input_length: Size of the array
    left_branch_embedding = Embedding(7000, 300, input_length=100)(left_branch)
    lstm_out = LSTM(256)(left_branch_embedding)
    lstm_out = Dropout(0.7)(lstm_out)
    lstm_out = Dense(128, activation='sigmoid')(lstm_out)

    right_branch = Input((7012, ))
    merged = Concatenate()([lstm_out, right_branch])
    output_layer = Dense(3, activation = 'softmax')(merged)

    model = Model(inputs=[left_branch, right_branch], outputs=output_layer)
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])