Which features do I select from text?

Question

Hello, I am very new to data science, machine learning, and stack overflow. Excuse me for being unclear or asking naive questions.

My question is as follows:

From any given document, I am trying to classify it according to the emotions it evokes in readers, using a neural network. However, I am having difficulties with feature selection. I'm thinking of using NLTK and RAKE to extract keywords, but I don't know how I can translate them into features. Should I hash the keywords for one feature? Or, should I find a dictionary of english words (i.e. Wordnet), and use every word in the dictionary as a feature.

Answer 1

Using NLTK in python you should first Tokenize the sentences into words, even you can use Ngram for 2-Gram or 3-Gram bags of word, the reason I am suggesting N-Gram is that let's suppose you have sentence like: I am not happy with this product, then 2-Gram tokenize it as ['not happy', 'happy with', 'with this', 'this product'] here I and am are assumed as STOPWORDS. Using HashingTF you can hash the sentence into a feature vector as ['word position': frequency of word, ...] i.e highly sparse vectors, For Hashing in PySpark check this documentation.

Here below python code will help you to tokenize in bags of word

import string

from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer

PUNCTUATION = set(string.punctuation)
STOPWORDS = set(stopwords.words('english'))
STEMMER = PorterStemmer()

example = ['Hello Krishna Prasad, this is test file for spark testing',
            'Another episode of star',
            'There are far and away many stars'
            'A galloping horse using two coconuts'
            'My kingdom for a horse'
            'A long time ago in a galaxy far']

def tokenize(text):
    tokens = word_tokenize(text)
    lowercased = [t.lower() for t in tokens]
    no_punctuation = []
    for word in lowercased:
        punct_removed = ''.join([letter for letter in word if not letter in PUNCTUATION])
        no_punctuation.append(punct_removed)
    no_stopwords = [w for w in no_punctuation if not w in STOPWORDS]
    stemmed = [STEMMER.stem(w) for w in no_stopwords]
    return [w for w in stemmed if w]

tokenized_word = [tokenize(text) for text in example]
for word in tokenized_word:
    print word

Out of the above code as:

$python WordFrequencyHash.py
[u'hello', u'krishna', u'prasad', u'test', u'file', u'spark', u'test']
[u'anoth', u'episod', u'star']
[u'far', u'away', u'mani', u'starsa', u'gallop', u'hors', u'use', u'two', u'coconutsmi', u'kingdom', u'horsea', u'long', u'time', u'ago', u'galaxi', u'far']

You can also use word2vec or countvectorizer for tokenization.

Answer 2

A feature extractor is just a function that returns the value of a feature given a target instance. For example, given in the input sentence "I friggin hate regex", you could run a tokenizer on it to break it into a list of words. You could then have a feature extractor function "hasCurseWord(tokens)" that returns true or false indicating presence of a curse word (you could have a dict of predefined curse words you compare to. Similarly you could write a feat extractor that numCurseWords that returns the count of curse words in a text. By analogy you can do the same thing with positive and negative words.

So in answer to your query, in addition to individual grams and perhaps phrasal ngrams (as mentioned by another answer), you will want to add custom feature extractors - e.g. curse words are good indicators of sentiment - to increase the accuracy of your sentiment classifier.

Here is what a feature extractor that counts developer-defined negative words might look like in Python:

def featx_negative(tokens, args=None):
       #negative terms
    num_neg = 0
    num_neg += len([w for w in tokens if re.search('hate', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('stuck', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('smh', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('angry', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('mad', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('blow', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('trash', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('garbage', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('bad', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('worst', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('<<<', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('dead', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('die', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('boo', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('horrib', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('terrib', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('annoy', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('wrong', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('dump', w, re.IGNORECASE)])
    num_neg += len([w for w in tokens if re.search('mess', w, re.IGNORECASE)])
    features = {}
    features['Has(NEGATIVE)'] = True if num_neg > 0 else False
    return features

The above approach, is what might be called a dictionary-approach -- essentially each feature extractor compares an input sentence to an externally defined dictionary of terms. The limitations of this approach are that you must manually create the dictionaries themselves (or find some external data source of positive/negative words). In addition, the dictionaries are static (what happens when a new slang term comes into existence?) and they essentially weight terms equally. The alternative approach would be to use a machine-learning algorithm that gets trained on sentences pre-labelled by hand (as pos/neg). This approach allows you to weight your words according to their actual distribution (e.g., tf-idf) or chi-sq. For further details, see this answer to creating training data.