I'm trying to really understand Tokenizing and Vectorizing text in machine learning, and am looking really hard into the Keras Tokenizer class. I get the mechanics of how it's used, but I'd like to really know more about it. So for example, there's this common usage:
tokens = Tokenizer(num_words=SOME_NUMBER)
tokens returns a word_index, which maps words to some number. Are the words all words in texts, or are they maxed at SOME_NUMBER? And are the dict values for word_index the frequency of each word, or just the order of the word?