3
$\begingroup$

I'm trying to really understand Tokenizing and Vectorizing text in machine learning, and am looking really hard into the Keras Tokenizer class. I get the mechanics of how it's used, but I'd like to really know more about it. So for example, there's this common usage:

tokens = Tokenizer(num_words=SOME_NUMBER)
tokens.fit_on_texts(texts)

tokens returns a word_index, which maps words to some number. Are the words all words in texts, or are they maxed at SOME_NUMBER? And are the dict values for word_index the frequency of each word, or just the order of the word?

$\endgroup$
2
  • $\begingroup$ word_index is used to find the length of the vector made. Use this the determine the extent of you vocabulary. $\endgroup$ Aug 5, 2021 at 16:54
  • $\begingroup$ in simpler words, its the number of unique tokens. $\endgroup$ Oct 21, 2021 at 15:27

1 Answer 1

1
$\begingroup$

Are the words all words in texts, or are they maxed at SOME_NUMBER?

Yes, it will be maxed at the most common SOME_NUMBER-1 words.

And are the dict values for word_index the frequency of each word, or just the order of the word?

It is just the index of the word in the dictionary.

You can read more here in the documentation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.