I had used TfidfVectorizer and passed it through MultinomialNB for document classification, It was working fine.

But now I need to pass huge set of documents for ex above 1 Lakh and when I am trying to pass theses document content to TfidfVectorizer my local computer hangged. It seems it has performance issue. So I got suggestion to use HashingVectorizer.

And I used below code for classification(Just replacing TfidfVectorizer by HashingVectorizer)

stop_words = open("english_stopwords").read().split("\n")
vect = HashingVectorizer(stop_words=stop_words, ngram_range=(1,5))
X_train_dtm = vect.fit_transform(training_content_list)
X_predict_dtm = vect.transform(predict_content_list)
nb = MultinomialNB()
nb.fit(X_train_dtm, training_label_list)
predicted_label_list = nb.predict(X_predict_dtm)

Got error:

File "/home/rajesh/www/rajesh/docuchief2/project/web/env/lib/python3.6/site-packages/sklearn/naive_bayes.py", line 720, in _count raise ValueError("Input X must be non-negative") ValueError: Input X must be non-negative

So I got TfidfVectorizer is calculated as per occurance of words so it works, but HashingVectorizer logic is differenct which i can not figure out how HashingVectorizer will implement in MultinomialNB.

Can someone please help me how I can solve this performance issue like.. Can I use TfidfVectorizer for huge training dataset if yes then how? If not then how can I use HashingVectorizer here?


You need to ensure that the hashing vector doesn't purpose negatives. The way to do this is via HashingVectorizer(non_negative=True).

  • $\begingroup$ After added non_negative=True, I got error TypeError: __init__() got an unexpected keyword argument 'non_negative'. My include package is from sklearn.feature_extraction.text import HashingVectorizer. $\endgroup$ – Rajesh das Sep 17 at 5:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.