NLP Emotion Detection - Model fails to learn to recognize negations

Question

I am working on a nlp emotion detection project. The emotions that I try to predict are 'joy', 'fear', 'anger', 'sadness'. I use some publicly available labeled datasets to train my model e.g. ISEAR, WASSA etc. I have tried the following approaches:

1. Traditional ML approached using bigrams and trigrams.
2. CNN with the following architecture: (X) Text -> Embedding (W2V pretrained on wikipedia articles) -> Deep Network (CNN 1D) -> Fully connected (Dense) -> Output Layer (Softmax) -> Emotion class (Y)
3. LSTM with the following architecture: (X) Text -> Embedding (W2V pretrained on wikipedia articles) -> Deep Network (LSTM/GRU) -> Fully connected (Dense) -> Output Layer (Softmax) -> Emotion class (Y)

The NN models achieve more than 80% accuracy but still when I use the trained model to predict the emotion on text that includes some negation I get the wrong results. For example:

Text : "I am happy with easy jet, it is a great company!"

Predicts Happy

Text: I am not happy with easyjet #unhappy_customer

Predicts Happy

Any suggestions on how to overcome this problem?

Answer 1

In general this is a difficult problem, it's about the problem of Natural Language Understanding which is far from being solved.

The advanced option requires a full syntactic parsing of the sentence, ideally followed by some kind of semantic representation of the sentence, for example by extracting relations. As far as I know this is rarely used because these steps will often cause as many errors as they solve.

Some more reasonable heuristics can be considered, for instance detecting specific negation words and either including this information as feature or modifying the original features accordingly (e.g. when a negation is detected replace the token "happy" with "not(happy)" in the features).

Note that it's unlikely to be perfect anyway, due to the usual obstacles: hedging ("I would assume that it was quite good"), sarcasm ("Sure, I was very happy with the terrible service"), metaphors ("I was feeling like a fish in water"), etc.