0
$\begingroup$

I notice in many of the tutorials 1 is added to the word_index. For example considering a sample code snippet inspired from Tensorflow's tutorial for NMT https://www.tensorflow.org/tutorials/text/nmt_with_attention :

import tensorflow as tf
sample_input = ["sample sentence 1", "sample sentence 2"]
lang_tokenizer = tf.keras.preprocessing.text.Tokenizer(filters='')
lang_tokenizer.fit_on_texts(sample_input)
vocab_inp_size = len(lang_tokenizer.word_index)+1

I dont understand the reason for adding 1 to the word_index dictionary. Wont adding a random 1 affect the prediction. Any suggestions will be helpful

$\endgroup$
4
  • $\begingroup$ Wondering if this is the additional dummy word to replace anything that is out-of-vocabulary $\endgroup$ – Jayaram Iyer Apr 28 at 4:13
  • $\begingroup$ Note that they are just adding 1 to the size of the vocabulary, not to the token IDs themselves. $\endgroup$ – noe Apr 28 at 6:44
  • $\begingroup$ @noe wow I just realized that. Ok, so it wont affect the prediction then. But can you please explain what is the reason to add 1? $\endgroup$ – data_person Apr 28 at 6:56
  • $\begingroup$ I added an answer with all the information. $\endgroup$ – noe Apr 28 at 8:06
3
$\begingroup$

First, note that they are just adding 1 to the size of the vocabulary, not to the token IDs themselves, so the predictions are not affected.

Then, why adding 1 ?

Because Tokenizer.word_index is a python dictionary that contains token keys (string) and token ID values (integer), and where the first token ID is 1 (not zero) and where the token IDs are assigned incrementally. Therefore, the greatest token ID in word_index is len(word_index). Therefore, we need vocabulary of size len(word_index) + 1 to be able to index up to the greatest token ID.

Update: note that adding 1 to the vocabulary size has nothing to do with out of vocabulary words: the words that are not pretrained are encoded as the out-of-vocabulary token (oov_token) if it was provided when building the tokenizer, or ignored if not. The oov token, if provided, has index 1.

$\endgroup$
5
  • $\begingroup$ Thanks. This is great. One confusion - I read in a tutorial https://stackabuse.com/python-for-nlp-word-embeddings-for-deep-learning-in-keras/ that the reason to add 1 is : Remember to add 1 with the vocabulary size. This is to store the dimensions for the words for which no pretrained word embeddings exist.. So does adding 1 serve the dual purpose? $\endgroup$ – data_person Apr 28 at 21:57
  • $\begingroup$ No, that is not correct. The words that are not pretrained are encoded as the out-of-vocabulary token (oov_token) if it was provided when building the tokenizer, or ignored if not. The oov token, if provided, has index 1. Adding 1 to the vocabulary size has nothing to do with out of vocabulary words. $\endgroup$ – noe Apr 29 at 6:30
  • $\begingroup$ I updated the answer to include this information. $\endgroup$ – noe Apr 29 at 6:32
  • $\begingroup$ Please consider upvoting the question if you considered it useful and also marking it as correct if deemed so. $\endgroup$ – noe Apr 29 at 6:32
  • $\begingroup$ Thanks, I have accepted the solution. You have mentioned to upvote the question, but cannot upvote my own question, please do so if you find the question useful to other learners. $\endgroup$ – data_person Apr 30 at 2:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.