Return to Question

deleted 81 characters in body

Source Link

edited Dec 7, 2022 at 17:59

1.7k
9
24
39

Good morning, first of all I would like to thank you in advance for your answer. I am doing an essay on the mathematics behind a text classifier with NLP and neural networks and I would like to know how exactly the TOKENIZER function of kerasKeras works. Whether cosine similarity is involved and how the dictionary creation is carried out taking frequency into account. If anyone knows the answer or a book/article where it is reflected, I will be eternally grateful.

MAX_NB_WORDS = 50000
MAX_SEQUENCE_LENGTH = 250
tokenizer = Tokenizer(num_words=MAX_NB_WORDS, filters='!"#$%&()*+,-./:;<=>?@[\]^_`{|}~', lower=True)
tokenizer.fit_on_texts(data['Consumer complaint narrative'].values)

Good morning, first of all I would like to thank you in advance for your answer. I am doing an essay on the mathematics behind a text classifier with NLP and neural networks and I would like to know how exactly the TOKENIZER function of keras works. Whether cosine similarity is involved and how the dictionary creation is carried out taking frequency into account. If anyone knows the answer or a book/article where it is reflected, I will be eternally grateful.

MAX_NB_WORDS = 50000
MAX_SEQUENCE_LENGTH = 250
tokenizer = Tokenizer(num_words=MAX_NB_WORDS, filters='!"#$%&()*+,-./:;<=>?@[\]^_`{|}~', lower=True)
tokenizer.fit_on_texts(data['Consumer complaint narrative'].values)

I am doing an essay on the mathematics behind a text classifier with NLP and neural networks and I would like to know how exactly the TOKENIZER function of Keras works. Whether cosine similarity is involved and how the dictionary creation is carried out taking frequency into account. If anyone knows the answer or a book/article where it is reflected, I will be eternally grateful.

MAX_NB_WORDS = 50000
MAX_SEQUENCE_LENGTH = 250
tokenizer = Tokenizer(num_words=MAX_NB_WORDS, filters='!"#$%&()*+,-./:;<=>?@[\]^_`{|}~', lower=True)
tokenizer.fit_on_texts(data['Consumer complaint narrative'].values)

Source Link

asked Dec 7, 2022 at 9:21

sara

What is the math behind the Keras Tokenizer() function?

Good morning, first of all I would like to thank you in advance for your answer. I am doing an essay on the mathematics behind a text classifier with NLP and neural networks and I would like to know how exactly the TOKENIZER function of keras works. Whether cosine similarity is involved and how the dictionary creation is carried out taking frequency into account. If anyone knows the answer or a book/article where it is reflected, I will be eternally grateful.

MAX_NB_WORDS = 50000
MAX_SEQUENCE_LENGTH = 250
tokenizer = Tokenizer(num_words=MAX_NB_WORDS, filters='!"#$%&()*+,-./:;<=>?@[\]^_`{|}~', lower=True)
tokenizer.fit_on_texts(data['Consumer complaint narrative'].values)