I had used TfidfVectorizer and passed it through MultinomialNB for document classification, It was working fine.
But now I need to pass huge set of documents for ex above 1 Lakh and when I am trying to pass theses document content to TfidfVectorizer my local computer hangged. It seems it has performance issue. So I got suggestion to use HashingVectorizer.
And I used below code for classification(Just replacing TfidfVectorizer by HashingVectorizer)
stop_words = open("english_stopwords").read().split("\n") vect = HashingVectorizer(stop_words=stop_words, ngram_range=(1,5)) X_train_dtm = vect.fit_transform(training_content_list) X_predict_dtm = vect.transform(predict_content_list) nb = MultinomialNB() nb.fit(X_train_dtm, training_label_list) predicted_label_list = nb.predict(X_predict_dtm)
File "/home/rajesh/www/rajesh/docuchief2/project/web/env/lib/python3.6/site-packages/sklearn/naive_bayes.py", line 720, in _count raise ValueError("Input X must be non-negative") ValueError: Input X must be non-negative
So I got TfidfVectorizer is calculated as per occurance of words so it works, but HashingVectorizer logic is differenct which i can not figure out how HashingVectorizer will implement in MultinomialNB.
Can someone please help me how I can solve this performance issue like.. Can I use TfidfVectorizer for huge training dataset if yes then how? If not then how can I use HashingVectorizer here?