0
$\begingroup$

I was following this example online for simple text classification

And when I create the classifier object like this

from sklearn.datasets import fetch_20newsgroups
twenty_train = fetch_20newsgroups(subset='train', shuffle=True)

from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(twenty_train.data)
X_train_counts.shape

from sklearn.feature_extraction.text import TfidfTransformer
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
X_train_tfidf.shape

from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB().fit(X_train_tfidf, twenty_train.target)

and run predict on the test data it gives an error

import numpy as np
twenty_test = fetch_20newsgroups(subset='test', shuffle=True)
predicted = clf.predict(twenty_test.data)
np.mean(predicted == twenty_test.target)

ValueError: Expected 2D array, got 1D array instead:

But when I do the same thing using Pipeline it works as in the example

from sklearn.pipeline import Pipeline
text_clf = Pipeline([('vect', CountVectorizer()),
('tfidf', TfidfTransformer()),
('clf', MultinomialNB())])

text_clf = text_clf.fit(twenty_train.data, twenty_train.target)
twenty_test = fetch_20newsgroups(subset='test', shuffle=True)
predicted = text_clf.predict(twenty_test.data)
np.mean(predicted == twenty_test.target)

Out[37]: 0.7738980350504514

Why is that?

$\endgroup$
1
$\begingroup$

It seems to be because the predict method on your Pipeline object requires the input to match the input of the first object in your pipeline, which is the CountVectorizer. It any case, it only requires an iterable object, which your 1d array indeed is.

The classifier you train without the pipeline ends up being a MultinomialNB object, whose predict method actually requires a 2d array of shape num_samples * num_features.

Maybe you need to pass your test data through each of the individual steps manually before feeding anything into the final classifier?

$\endgroup$
0
$\begingroup$

I met the same problem as you. After I read the related documents, I found the problem is that the main idea of Pipeline. We do CountVector then feed into TfidfTransformer, then we do train or predict. But we have API TfidfVectorizer(). You definitely take the TfidfTransformer() as TfidfVectorizer().

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.