# Reason for adding 1 to word index for sequence modeling

I notice in many of the tutorials 1 is added to the word_index. For example considering a sample code snippet inspired from Tensorflow's tutorial for NMT https://www.tensorflow.org/tutorials/text/nmt_with_attention :

import tensorflow as tf
sample_input = ["sample sentence 1", "sample sentence 2"]
lang_tokenizer = tf.keras.preprocessing.text.Tokenizer(filters='')
lang_tokenizer.fit_on_texts(sample_input)
vocab_inp_size = len(lang_tokenizer.word_index)+1


I dont understand the reason for adding 1 to the word_index dictionary. Wont adding a random 1 affect the prediction. Any suggestions will be helpful

• Wondering if this is the additional dummy word to replace anything that is out-of-vocabulary – Jayaram Iyer Apr 28 at 4:13
• Note that they are just adding 1 to the size of the vocabulary, not to the token IDs themselves. – noe Apr 28 at 6:44
• @noe wow I just realized that. Ok, so it wont affect the prediction then. But can you please explain what is the reason to add 1? – data_person Apr 28 at 6:56
• I added an answer with all the information. – noe Apr 28 at 8:06

First, note that they are just adding 1 to the size of the vocabulary, not to the token IDs themselves, so the predictions are not affected.

Because Tokenizer.word_index is a python dictionary that contains token keys (string) and token ID values (integer), and where the first token ID is 1 (not zero) and where the token IDs are assigned incrementally. Therefore, the greatest token ID in word_index is len(word_index). Therefore, we need vocabulary of size len(word_index) + 1 to be able to index up to the greatest token ID.
• Thanks. This is great. One confusion - I read in a tutorial https://stackabuse.com/python-for-nlp-word-embeddings-for-deep-learning-in-keras/ that the reason to add 1 is : Remember to add 1 with the vocabulary size. This is to store the dimensions for the words for which no pretrained word embeddings exist.. So does adding 1 serve the dual purpose? – data_person Apr 28 at 21:57