I'm using StanfordNLP (java pakage) for twitter sentiment analysis. I want to extract keywords from the tweets, classify the keywords as positive, negative, or neutral, and use those counts to classify the polarity of the tweet.
How can this be done in Java? Please share few examples. What is the most used algorithm for keyword extraction?
The OP asks two different questions: (1) how to extract key words and (2) how to assign keywords a sentiment class (pos/neg/neu). I will address the keyword identification piece in this answer as many others have discussed how to do sentiment analysis (e.g., this post).
The approach I would suggest is a key keyword approach advocated by Mike Scott (author of WordSmith Tools) and Chris Tribble (a computational linguist). As discussed here, the basic approach is to create two corpora, your target corpus which is composed of texts sampled from the texts you are interested in, and a reference corpus, which is typically a much larger corpus (which is often more general in content).
The procedure begins by computing word (or n-gram) frequencies for both corpora. During this process, if a word’s frequency in the target corpus is found to be statistically probable in comparison to the reference corpus (as computed by a chi-squared test and a user defined p-value), it is considered a keyword (Baker, 2004). According to Scott (2006), the process typically identifies three types of words as key: proper nouns, words that characterize a text’s “aboutness,” and high frequency words that are indicators of style or genre. I discuss the approach in greater length in this 2007 article where I use the method to extract the salient features of academic discourse.
To illustrate with a concrete example, imagine your are interested in identifying the key topics surrounding a target brand (say, Pepsi Cola) as expressed in social media (e.g., twitter). Create two corpora: for your target corpora you create a search for "pepsi" and for the reference corpora you could search for competitor brands of soft drinks ("coke", "mountain dew", "dr. pepper", etc,). When the keyword process has terminated you will be left with all the keywords/topics that differentiate pepsi from other soft drink brands (and as a bonus you will also identify negative keywords... words that occur statistically less frequent in the target corpus).
As you might surmise, the results you get depend upon how the reference corpus is constructed. This, in my opinion, is a feature - as it gives the researcher much greater flexibility in hypothesis testing and data exploration.
I'm sorry I'm not a Java user and I never worked with StanfordNLP.
But I do know that the gini impurity criterion with and without decision tree were successfully applied for text classification. Moreover these tools have the ability to let you easily understand which features ( i.e. words in your case) contribute to the decision.
I know that convolutional neural network are a lot apply to sentiment analysis but you will have a really hard time trying to understand the intern-behaviour of the network.
I know this is not really an answer but I hope it might help :)
One easy solution is to prepare a emotion dictionary first. Such dictionary can be found online easily such as http://www.psychpage.com/learning/library/assess/feelings.html.
A simple workflow will be as follow:
Although this approach does not use any machine learning, the result is quite good. Another benefit is not need to prepare the labeled data set.
Of course, we should keep checking any missing emotion vocabulary.
The workflow of checking missing emotion
If I'd have to do that, I would use the "term-frequency inverse-document-frequency": https://en.wikipedia.org/wiki/Tf%E2%80%93idf.
BTW: I don't know how familiar you are with machine learning but it could be a good workflow.
