How to use different classes of words in CountVectorizer()

Question

Suppose I have a piece of writing and I want to assign probabilities to different genres (classes) based on its contents. For example

Text #1 : Comedy 10%, Horror 50%, Romance 1%

Text #2 : Comedy 40%, Horror 3%, Romance 30%

We have given keywords in each class through which we make a comparison. Below is the code that explains this scenario better

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Comedy
keywords_1 = ['funny', 'amusing', 'humorous', 'hilarious', 'jolly']

# Horror
keywords_2 = ['horror', 'fear', 'shock', 'panic', 'scream']

# Romance
keywords_3 = ['romantic', 'intimate', 'passionate', 'love', 'fond']

text = ('funny hilarious fear passionate')

cv1 = CountVectorizer(vocabulary = keywords_1) 
data = cv1.fit_transform([text]).toarray()
vec1 = np.array(data)
vec2 = np.array([[1, 1, 1, 1, 1]])
print(cosine_similarity(vec1, vec2))

cv2 = CountVectorizer(vocabulary = keywords_2) 
data = cv2.fit_transform([text]).toarray()
vec1 = np.array(data)
vec2 = np.array([[1, 1, 1, 1, 1]])
print(cosine_similarity(vec1, vec2))

cv3 = CountVectorizer(vocabulary = keywords_3) 
data = cv3.fit_transform([text]).toarray()
vec1 = np.array(data)
vec2 = np.array([[1, 1, 1, 1, 1]])
print(cosine_similarity(vec1, vec2))

The problem with this approach is that vocabulary in CountVectorizer() doesn't consider different word classes (Nouns, Verbs, Adjectives, Adverbs, plurals, etc.) of a word in a text. For example, let's say we have keywords list as below

keywords_1 = [(...), ('amusement', 'amusements', 'amuse', 'amuses', 'amused', 'amusing'), (...), ('hilarious', 'hilariously') (...)]

and we want to compute similarity as follows

cv1 = CountVectorizer(vocabulary = keywords_1) 
data = cv1.fit_transform([text]).toarray()
vec1 = np.array(data) # [[f1, f2, f3, f4, f5]]) # fi is the count of number of keywords matched in a sublist
vec2 = np.array([[n1, n2, n3, n4, n5]]) # ni is the size of sublist
print(cosine_similarity(vec1, vec2))

How can we modify the above code to capture this scenario. Any advice is appreciated.

Answer 1

First of all your question is about stemming words as mentioned in the other answer which can be found in any Python NLP library such as Spacy or NLTK.

The other point to mention here is that despite the other answer, what libraries has as Stop Words list is not actually stop word! Do no remove them! In NLP stop words should be extracted based on working corpus not based on a predefined list. In practice removing this kind of stop words usually reduces the performance on specific domain corpuses.

The Third point is that depending on the classifier and loss function you use, TF-IDF might be better than Count Vectorizer. I suppose it works better specially if Log Loss is the cost function but I am not sure. Just give it a try.

Answer 2

If you want to use for vocabulary sets like your example:

keywords_1 = [
    (...), 
    ('amusement', 'amusements', 'amuse', 'amuses', 'amused', 'amusing'),
    (...), 
    ('hilarious', 'hilariously'),
    (...)
]

I advise to you use stemmer and delete stopword before it.