1
$\begingroup$

I had used TfidfVectorizer and passed it through MultinomialNB for document classification, It was working fine.

But now I need to pass a huge set of documents for ex above 1 Lakh and when I am trying to pass these document content to TfidfVectorizer my local computer hanged. It seems it has a performance issue. So I got a suggestion to use HashingVectorizer.

And I used below code for classification(Just replacing TfidfVectorizer by HashingVectorizer)

stop_words = open("english_stopwords").read().split("\n")
vect = HashingVectorizer(stop_words=stop_words, ngram_range=(1,5))
X_train_dtm = vect.fit_transform(training_content_list)
X_predict_dtm = vect.transform(predict_content_list)
nb = MultinomialNB()
nb.fit(X_train_dtm, training_label_list)
predicted_label_list = nb.predict(X_predict_dtm)

Got error:

File "/home/rajesh/www/rajesh/docuchief2/project/web/env/lib/python3.6/site-packages/sklearn/naive_bayes.py", line 720, in _count raise ValueError("Input X must be non-negative") ValueError: Input X must be non-negative

So I got TfidfVectorizer is calculated as per occurrence of words so it works, but HashingVectorizer logic is different which I can not figure out how HashingVectorizer will implement in MultinomialNB.

Can someone please help me with how I can solve this performance issue. Can I use TfidfVectorizer for a huge training dataset if yes then how? If not then how can I use HashingVectorizer here?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
$\begingroup$

You need to ensure that the hashing vector doesn't purpose negatives. The way to do this is via HashingVectorizer(non_negative=True).

Improve this answer
$\endgroup$
2
  • $\begingroup$ After added non_negative=True, I got error TypeError: __init__() got an unexpected keyword argument 'non_negative'. My include package is from sklearn.feature_extraction.text import HashingVectorizer. $\endgroup$
    – Rajesh das
    Commented Sep 17, 2019 at 5:53
  • $\begingroup$ @Rajeshdas are you able to resolve the issue? $\endgroup$
    – devanathan
    Commented May 11, 2020 at 11:34
1
$\begingroup$

It doesn't seem that non_negative is an argument in some versions. Try using decode_error = 'ignore'. If you're working with a large dataset, this error could also be resulting from hash collisions, which can be solved by increasing the number of features:

vect = HashingVectorizer(decode_error = 'ignore',
                        n_features = 2**21,
                        preprocessor = None)
Improve this answer
$\endgroup$
1
$\begingroup$

In newer versions of sklearn, use HashingVectorizer(alternate_sign=False)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.