0
$\begingroup$
# create a count vectorizer object
count_vectorizer = CountVectorizer()

# fit the count vectorizer using the text data
count_vectorizer.fit(data['text'])

# collect the vocabulary items used in the vectorizer
dictionary = count_vectorizer.vocabulary_.items()

To my understanding, after count_vectorizer fits to data['text'], it generates a list of features. In my case, it generated 25,257 features and these are mapped as dict data type when I call count_vectorizer.vocabulary_. Which is still 25,257 tuples. It means, it used all the features.

Problem is, when I call count_vectorizer.vocabulary_.items() it returns 15,142 tuples as dict_items. Why the number has been reduced here? Should't all the features be used to make the dictionary?

Here are the lengths I'm talking about:

len(data['text'])  #19579
len(count_vectorizer.get_feature_names())  #25257 items
len(count_vectorizer.vocabulary_)  #25257 items
len(dictionary) #15142 items (??????)
$\endgroup$
  • $\begingroup$ Interesting. The mismatch in the last two lines suggests that something weird is happening at the python level, not necessarily due to sklearn. Why/when is len(my_dict) not equal to len(my_dict.items())??? $\endgroup$ – Ben Reiniger Mar 26 at 19:43
  • 1
    $\begingroup$ I tried with a dataset of mine and I have equal results. Are you sure it has nothing to do with your dataset or working session? $\endgroup$ – Edoardo Guerriero Mar 26 at 19:47
  • $\begingroup$ @BenReiniger I am asking the same question as you. As there are lots of features I am not sure which features are missing. $\endgroup$ – Ashraful Alam Imran Mar 26 at 19:50
  • 1
    $\begingroup$ Try taking the difference? Along @EdoardoGuerriero's lines, are you running these in immediate succession? $\endgroup$ – Ben Reiniger Mar 26 at 19:52
  • 2
    $\begingroup$ @EdoardoGuerriero called it as a session problem; care to make that an answer? $\endgroup$ – Ben Reiniger Mar 26 at 21:18
1
$\begingroup$

For future readers: as Ben wrote in the comments, it is really hard in general that len(my_dict) != len(my_dict.items()). When these kind of strage behaviours happen, it is always a good practise to perform some routine checks:

  • Clean your environment from every variable, even better restart the kernel and then run again your code.

  • Check your code for variables with same name, be sure you're not overwriting stuff, it's easy also to end up defining functions with same name of other imported library functions.

  • Try your code on different datasets, to be sure the mistake is not due to some external issue with the data.

If none of the above checks solve the mystery, then you can start thinking about a bug, a corrupted installation of a library or some other more nested problems.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.