I'm using StanfordNLP (java pakage) for twitter sentiment analysis. I want to extract keywords from the tweets, classify the keywords as positive, negative, or neutral, and use those counts to classify the polarity of the tweet.

How can this be done in Java? Please share few examples. What is the most used algorithm for keyword extraction?


The OP asks two different questions: (1) how to extract key words and (2) how to assign keywords a sentiment class (pos/neg/neu). I will address the keyword identification piece in this answer as many others have discussed how to do sentiment analysis (e.g., this post).

The approach I would suggest is a key keyword approach advocated by Mike Scott (author of WordSmith Tools) and Chris Tribble (a computational linguist). As discussed here, the basic approach is to create two corpora, your target corpus which is composed of texts sampled from the texts you are interested in, and a reference corpus, which is typically a much larger corpus (which is often more general in content).

The procedure begins by computing word (or n-gram) frequencies for both corpora. During this process, if a word’s frequency in the target corpus is found to be statistically probable in comparison to the reference corpus (as computed by a chi-squared test and a user defined p-value), it is considered a keyword (Baker, 2004). According to Scott (2006), the process typically identifies three types of words as key: proper nouns, words that characterize a text’s “aboutness,” and high frequency words that are indicators of style or genre. I discuss the approach in greater length in this 2007 article where I use the method to extract the salient features of academic discourse.

To illustrate with a concrete example, imagine your are interested in identifying the key topics surrounding a target brand (say, Pepsi Cola) as expressed in social media (e.g., twitter). Create two corpora: for your target corpora you create a search for "pepsi" and for the reference corpora you could search for competitor brands of soft drinks ("coke", "mountain dew", "dr. pepper", etc,). When the keyword process has terminated you will be left with all the keywords/topics that differentiate pepsi from other soft drink brands (and as a bonus you will also identify negative keywords... words that occur statistically less frequent in the target corpus).

As you might surmise, the results you get depend upon how the reference corpus is constructed. This, in my opinion, is a feature - as it gives the researcher much greater flexibility in hypothesis testing and data exploration.


I'm sorry I'm not a Java user and I never worked with StanfordNLP.

But I do know that the gini impurity criterion with and without decision tree were successfully applied for text classification. Moreover these tools have the ability to let you easily understand which features ( i.e. words in your case) contribute to the decision.

I know that convolutional neural network are a lot apply to sentiment analysis but you will have a really hard time trying to understand the intern-behaviour of the network.

I know this is not really an answer but I hope it might help :)


One easy solution is to prepare a emotion dictionary first. Such dictionary can be found online easily such as http://www.psychpage.com/learning/library/assess/feelings.html.

A simple workflow will be as follow:

  1. For each tweet, tokenize the tweet into a list of vocabularies
  2. For each list of vocabularies, count the (positive, neutral, negative) words
  3. Classify a tweet as (positive/ neutral/ negative) emotion by the counting

Although this approach does not use any machine learning, the result is quite good. Another benefit is not need to prepare the labeled data set.

Of course, we should keep checking any missing emotion vocabulary.

The workflow of checking missing emotion

  1. For each tweet, stem and tokenize by StandordNLP (ex. "Happy" will converted to ("HAPPY", "JJ") where the tag started from JJ refer to adjective
  2. Filter the tokens to extract the absent words which are absent in the prepared missing emotion vocabulary and has tag started with "JJ"
  3. Do the word count on the missing adjective and sort by frequency in reverse order
  4. Manually add the missing adjectives if necessary
  • $\begingroup$ can you please share a code example of your explanation @Icarus $\endgroup$ – Dilip Bobby Jun 28 '16 at 4:47
  • $\begingroup$ Do you prefer using spark with map-reduce or just a simple code example ? $\endgroup$ – Icarus Jun 28 '16 at 4:54
  • $\begingroup$ I'm familiar with spark with map-reduce and simple java code :) you can share both code examples.it will be best practice to me @ Icarus $\endgroup$ – Dilip Bobby Jun 30 '16 at 6:22

If I'd have to do that, I would use the "term-frequency inverse-document-frequency": https://en.wikipedia.org/wiki/Tf%E2%80%93idf.

  1. tokenize tweets of your topic
  2. calculate the tf-idf
  3. take all words with a high score of tf-idf and train a NN with it (as supervised machine learning) - use bag-of-words to get a good representation
  4. use the NN on new tweets

BTW: I don't know how familiar you are with machine learning but it could be a good workflow.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.