How does one go about feature extraction for training labelled tweets for sentiment analysis?

Question

I am currently working on a sentiment analysis project where the end goal is to try and predict who wins the Nigerian 2019 presidential election. My goal is to create a model that analyses tweets and detects the tone of a tweet (whether positive, negative or neutral). This model can in turn be used to predict who wins. Here's what I have done so far.

1. I have a program running on a server streaming tweets daily
2. I have labelled about 5000 tweets manually into different categories (positive, negative and neutral tweets).
3. I previously used text blob to determine sentiment on each tweet. But it was not accurate enough.
4. I was then advised to train a model on the way Nigerians speak on twitter, which can sometimes be a little different from normal English

From my research, I found out I have to use feature extraction before I can train the model with my labelled data. My problem now is that I don't understand how to go about these feature extraction. I have read materials about what people use (bag of words, bigrams, unigrams, part of speech tags etc). I don't understand the explanation I got so far, how it all comes together or which to use. Please could anyone explain these in layman's terms or point me to resources that explains this in layman's terms. Thank you

Answer 1

Feature Extraction is an important step when dealing with natural languages because the text you've collected isn't in a form understandable by a computer. If you have a tweet that goes something like

I do not like the views of @Candidate1 on #Topic1. Too conservative!! I can't stand it!

then we can't just feed this into a learning algorithm. We need to convert it into a proper format, so we perform pre-processing on our data.

To start, we might want to try to tokenize the tweet. To tokenize is something like segmentation. If you're familiar with Python, we can use ready made libraries (like NLTK) to aid us. Depending on how your tokenizer is made, you could transform the previous tweet into something like

['I', 'do', 'not', 'like', 'the', 'views', 'of', '@', 'Candidate1', 'on', '#', 'Topic1', '.', 'Too', 'conservative', '!', '!', 'I', 'can', "'", 't', 'stand', 'it', '!']

I manually segmented the tweet with the rule that words separated by a space should be tokens and punctuation should be tokenized separately. The way you build a tokenizer (or adjust the settings of a ready made one) will determine the output from a tweet. Notice how '@' and 'Candidate1' are separated? A tokenizer for regular text might not be able to identify that this is a social media entity -- a user mention. If you can adjust your tokenizer to account for social media identities and contractions (like "can't"), you could produce a list of tokens as such

['I', 'do', 'not', 'like', 'the', 'views', 'of', '@Candidate1', 'on', '#Topic1', '.', 'Too', 'conservative', '!', '!', 'I', "can't", 'stand', 'it', '!']

Now, you mentioned bigrams and unigrams. An n-grams (e.g. 1-gram == unigram) is just a sequence of tokens. So what we produced awhile ago was just a unigram list of tokens. If you want a bigram, then you'd want to take 2 tokens at a time. An example output of a bigram list of tokens would be

['I do', 'do not', 'not like', 'like the', 'the views', 'views of', 'of @Candaite1', '@Candidate1 on', 'on #Topic1', '#Topic1 .', '. Too', 'Too conservative', 'conservative !', '! !', '! I', "I can't", "can't stand", 'stand it', 'it !',]

Notice how words repeat? Imagine what a 3-gram or 5-gram would look like.

Before anything, why would we use bigrams over unigrams or those of higher n values? Well, the higher the n, the more about of order you're able to capture. Sometimes order is an important factor in learning. Playing around with how you'll represent your data might show you important features.

Now that we have our text tokenized, we can start extracting features! We can turning our example test, and other text samples, into a Bag-Of-Words (BOW) model. Think of a BOW as a table with column headers as the words/terms and rows as your text samples/tweets. A cell could then contain the number of words/terms for a given sample of text. You could start with counting each term in a sample, so based on the tweet, you'd come up with something like

tweet1: {
   'I': 2,
   'do': 1,
   'not': 1,
   'like': 1,
   'the': 1,
   ...
   '!': 3,
   ... 
}

I don't want to manually write it all, but I hope you get the picture here. See how 'I' is 2 because it was mentioned twice in the sample. '!' was mentioned trice, so its value was 2. You'll find that there will be a lot of 1 values, specially in tweets, because there isn't room for much to be written.

You'd do this for each of your tweets and you'll come up with something like

       | 'I' | '!' | '#Candiate1' | ...
tweet1 |  2  |  3  |       1
tweet2 |  0  |  0  |       0
...    
tweetn |  1  |  5  |       0

This is a BOW! Adjust your n-grams and you'll get different features with different values. This representation is actually a simple starting point for you because this is now understandable by a computer. You mentioned you labeled your tweets, so imagine that table, but with your label appended to it.

       | 'I' | '!' | '#Candiate1' | label
tweet1 |  2  |  3  |       1      | negative
tweet2 |  0  |  0  |       0      | negative
...    
tweetn |  1  |  5  |       0      | positive

And with that, you can build a simple model with the appropriate steps after feature extraction.

---

There is a lot more to feature extraction! So many different combinations of methods to try out, but stick to things simple for now, if you don't feel so adventurous. Here are some things to consider:

• Instead of counting terms, you could go with frequency (i.e. number of occurrence of a term divided by total number of terms)
• Instead of extracting the words themselves, you can use part-of-speech (POS) tags (POS tags are language dependent, so you'll have to look for a POS reference for the specific language you're dealing with)
• Consider extracting topic models or maybe meta information (avg length of words, avg length of tweets)
• Is there an emotion lexicon for the language you're dealing with? Consider counting the number of emotion related words
• Consider reducing the number of unique words or applying weighted schemes (like TFIDF)

There are so many ways to approach this, but it ultimately boils down to a design you'll finalize. I guess the best advice I can give you is to refer to related literature. Look at other papers/articles that process text written by Nigerians. What are the methods they use to process text? Is it language dependent or independent? Did they reduce the number of features? Can I create the language resources? Once you identify a good source, try modifying and experimenting from there.

Answer 2

I'll throw at you some standard names that the are you're building a project on has, every term that you don't understand, research it, it'll improve your knowledge a lot.

First, there are great resources out there on NLP (Natural Language Processing)regarding classification problems such as this one: http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html

Which algorithm to use? It depends on your data, there are two topics that you have to pay attention: the over or underfitting of your model and the distribution of your classes, those two things if managed correctly can make your model way better than those that don't look at it.

Try to explore your data, plot the distribution of the classes based on a single word, two words, a phrase, a hashtag, or something like that, try to see if they can describe one or more classes, your model will probably improving by doing that.

Here are two articles about feature extraction on text data:

https://towardsdatascience.com/machine-learning-nlp-text-classification-using-scikit-learn-python-and-nltk-c52b92a7c73a

https://towardsdatascience.com/multi-class-text-classification-with-scikit-learn-12f1e60e0a9f