0
$\begingroup$

I was starting an NLP project and simply get a "CountVectorizer()" output anytime I try to run CountVectorizer.fit on the list. I've had the same issue across multiple IDE's, and different code. I've looked online, and even copy and pasted other codes with their lists and I receive the same CountVectorizer() output.

My code is as follows:

cv = CountVectorizer()

messages = ['This is a good product', 'It was a bit pricey for what it does', 'I found good value here']

cv.fit(messages)
**output ----->  CountVectorizer()**

I'm really stumped on this issue. Any advice would be greatly appreciated. Thanks.

Update: This seems to be a local issue as I am able to get it to fit on Colab. If anyone can suggest what might be going on I'd be estatic.

$\endgroup$
1
$\begingroup$

This is normal, the fit method of scikit-learn'sCountverctorizer object changes the object inplace, and returns the object itself as explained in the documentation. This holds true for any sklearn object implementing a fit method, by the way.

When you run:

cv.fit(messages)

The cv object learns the transformation (here it is simply about counting words) and stores all the necessary parameters in order to be able to later apply the transformation to any new data point.

If you want to transform the data, you need to use first the fit method (to learn the transformation as said before) and then the transform method.

In your case if you want to transform your list of sentences you should do as follows:

cv = CountVectorizer()

messages = ['This is a good product', 'It was a bit pricey for what it does', 'I found good value here']

cv.fit(messages)
vectorized_messages = cv.transform(messages)

vectorized_messages

This will output:

<3x14 sparse matrix of type '<class 'numpy.int64'>'
    with 15 stored elements in Compressed Sparse Row format>

So you will have successfully transformed your list of strings to a a matrix of token counts.

Note that sklearn often also implement a fit_transform method that does bothe at the same time:

cv = CountVectorizer()
cv.fit(messages)
vectorized_messages = cv.transform(messages)

#is the same as:
cv = CountVectorizer()
vectorized_messages = cv.transform(messages)
$\endgroup$
6
  • $\begingroup$ I think I understand. However, I've seen it run in online competition notebooks, and they get something like the following,output I received with the exact same code in Colab: CountVectorizer(analyzer='word', binary=False, decode_error='strict', dtype=<class 'numpy.int64'>, encoding='utf-8', input='content', lowercase=True, max_df=1.0, max_features=None, min_df=1, ngram_range=(1, 1), preprocessor=None, stop_words=None, strip_accents=None, token_pattern='(?u)\\b\\w\\w+\\b', tokenizer=None, vocabulary=None) $\endgroup$ Dec 8 '20 at 14:18
  • $\begingroup$ Could you give me a link to such an example? $\endgroup$
    – A Co
    Dec 8 '20 at 14:19
  • $\begingroup$ kaggle.com/faressayah/… $\endgroup$ Dec 8 '20 at 14:21
  • $\begingroup$ I see. It does not change anything to the code and the way it behaves, this is really a matter of display. Maybe the difference is linked to the type of notebook, the version of sklearn and/or the version of python used. But it is really not important. What is printed in the notebook you linked are all the parameters of the CountVectorizer object. You can display them using print(cv.get_params()) $\endgroup$
    – A Co
    Dec 8 '20 at 14:28
  • 1
    $\begingroup$ Yes completely, the CountVectorizer() object has been fitted. You can check by calling cv.get_feature_names(): if the object has been fit this will output the different words used by the vectorizer, that should correspond to the words in the text you passed it. $\endgroup$
    – A Co
    Dec 8 '20 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.