count_vectorizer.vocabulary_.items() and count_vectorizer.vocabulary_ Inconsistent number of returns

Question

# create a count vectorizer object
count_vectorizer = CountVectorizer()

# fit the count vectorizer using the text data
count_vectorizer.fit(data['text'])

# collect the vocabulary items used in the vectorizer
dictionary = count_vectorizer.vocabulary_.items()

To my understanding, after count_vectorizer fits to data['text'], it generates a list of features. In my case, it generated 25,257 features and these are mapped as dict data type when I call count_vectorizer.vocabulary_. Which is still 25,257 tuples. It means, it used all the features.

Problem is, when I call count_vectorizer.vocabulary_.items() it returns 15,142 tuples as dict_items. Why the number has been reduced here? Should't all the features be used to make the dictionary?

Here are the lengths I'm talking about:

len(data['text'])  #19579
len(count_vectorizer.get_feature_names())  #25257 items
len(count_vectorizer.vocabulary_)  #25257 items
len(dictionary) #15142 items (??????)

Answer 1

For future readers: as Ben wrote in the comments, it is really hard in general that len(my_dict) != len(my_dict.items()). When these kind of strage behaviours happen, it is always a good practise to perform some routine checks:

• Clean your environment from every variable, even better restart the kernel and then run again your code.

• Check your code for variables with same name, be sure you're not overwriting stuff, it's easy also to end up defining functions with same name of other imported library functions.

• Try your code on different datasets, to be sure the mistake is not due to some external issue with the data.

If none of the above checks solve the mystery, then you can start thinking about a bug, a corrupted installation of a library or some other more nested problems.