# How to use different classes of words in CountVectorizer()

Suppose I have a piece of writing and I want to assign probabilities to different genres (classes) based on its contents. For example

Text #1 : Comedy 10%, Horror 50%, Romance 1%

Text #2 : Comedy 40%, Horror 3%, Romance 30%

We have given keywords in each class through which we make a comparison. Below is the code that explains this scenario better

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Comedy
keywords_1 = ['funny', 'amusing', 'humorous', 'hilarious', 'jolly']

# Horror
keywords_2 = ['horror', 'fear', 'shock', 'panic', 'scream']

# Romance
keywords_3 = ['romantic', 'intimate', 'passionate', 'love', 'fond']

text = ('funny hilarious fear passionate')

cv1 = CountVectorizer(vocabulary = keywords_1)
data = cv1.fit_transform([text]).toarray()
vec1 = np.array(data)
vec2 = np.array([[1, 1, 1, 1, 1]])
print(cosine_similarity(vec1, vec2))

cv2 = CountVectorizer(vocabulary = keywords_2)
data = cv2.fit_transform([text]).toarray()
vec1 = np.array(data)
vec2 = np.array([[1, 1, 1, 1, 1]])
print(cosine_similarity(vec1, vec2))

cv3 = CountVectorizer(vocabulary = keywords_3)
data = cv3.fit_transform([text]).toarray()
vec1 = np.array(data)
vec2 = np.array([[1, 1, 1, 1, 1]])
print(cosine_similarity(vec1, vec2))


The problem with this approach is that vocabulary in CountVectorizer() doesn't consider different word classes (Nouns, Verbs, Adjectives, Adverbs, plurals, etc.) of a word in a text. For example, let's say we have keywords list as below

keywords_1 = [(...), ('amusement', 'amusements', 'amuse', 'amuses', 'amused', 'amusing'), (...), ('hilarious', 'hilariously') (...)]

and we want to compute similarity as follows

cv1 = CountVectorizer(vocabulary = keywords_1)
data = cv1.fit_transform([text]).toarray()
vec1 = np.array(data) # [[f1, f2, f3, f4, f5]]) # fi is the count of number of keywords matched in a sublist
vec2 = np.array([[n1, n2, n3, n4, n5]]) # ni is the size of sublist
print(cosine_similarity(vec1, vec2))


How can we modify the above code to capture this scenario. Any advice is appreciated.

First of all your question is about stemming words as mentioned in the other answer which can be found in any Python NLP library such as Spacy or NLTK.

The other point to mention here is that despite the other answer, what libraries has as Stop Words list is not actually stop word! Do no remove them! In NLP stop words should be extracted based on working corpus not based on a predefined list. In practice removing this kind of stop words usually reduces the performance on specific domain corpuses.

The Third point is that depending on the classifier and loss function you use, TF-IDF might be better than Count Vectorizer. I suppose it works better specially if Log Loss is the cost function but I am not sure. Just give it a try.

• Hello @Kasra Manshaei, Is there a need to down-weight term frequency of keywords. TF-IDF is widely used for text classification but here our task is multi label Classification i.e to assign probabilities to different labels. I believe creating a TF vector by CountVectorizer() would work fine because here we are concerned more with presence or absence of keyword in a document rather than how important is it as compared to other documents. What do you think ? – Atinesh Jan 17 '18 at 17:50
• What I said was just a practical hint. I am dealing with exactly same problem at the moment and after 2 days we dropped CountVectorizer from all the codes! The magnitude of error was several times larger. About presence or absence of the word I would say there is no difference between them. TF-IDF just kind of normalizes the CounVectorizer. Probably the un-normalized nature of counting removes out many features just because "THEIR CLASSES" are small! not because they are not important. If this is the case, then normalized nature of TF-IDF helps. – Kasra Manshaei Jan 18 '18 at 10:07

If you want to use for vocabulary sets like your example:

keywords_1 = [
(...),
('amusement', 'amusements', 'amuse', 'amuses', 'amused', 'amusing'),
(...),
('hilarious', 'hilariously'),
(...)
]


I advise to you use stemmer and delete stopword before it.