Does this error means that the word doesn't exist in the tokenizer

return sent.split(" ").index(word)
ValueError: 'rat' is not in list

the code sequences like

def sentences():
   for sent in sentences:
       token = tokenizer.tokenize(sent)
       for i in token :
           idx = get_word_idx(sent,i)
def get_word_idx(sent: str, word: str):
    return sent.split(" ").index(word)

sentences split returns ['long', 'restaurant', 'table', 'with', 'rattan', 'rounded', 'back', 'chairs'] which rattan here is the problem as i think

  • $\begingroup$ The first return statement gives the index of a specific word in the sentence. The issue is simply that the list of words in the sentence does not contain the word 'rat' . $\endgroup$
    – Oxbowerce
    Jan 18, 2022 at 12:54
  • $\begingroup$ i think that because the tokenization tokenized the words so when it got rattan word it got rat which is doesn't exist in the list already .. should i add it as new_token ? as i tried to add rattan and the error is solved , does that right ? $\endgroup$
    – Begnnier
    Jan 18, 2022 at 13:04

1 Answer 1


First, a tokenizer doesn't have a dictionary of predefined words, so anyway it doesn't make sense to "add a new token" to a tokenizer.

Instead it uses indications in the text in order to separate the tokens. The most common indication is of course a whitespace character " ", but there are lots of cases where it's more complex than that. This is why there would be many cases where the second method with sent.split(" ").index(word) would not return the same tokens (punctuation marks, for example).

Also the tokenizer doesn't change the text, so if the sentence contains the word rattan it cannot transform it into the word rat. Why are you testing this? Btw rattan is a real word, in case this is the issue.

  • $\begingroup$ i tried to get embedding for the words in my file which contains sentences but got this error. excuse me can you help in this datascience.stackexchange.com/questions/107238/… $\endgroup$
    – Begnnier
    Jan 20, 2022 at 15:34
  • $\begingroup$ @Begnnier ok, you seem to have a confusion between the tokenizer and the pretrained word embedding model, these two are independent. it's totally normal than some words are not present in a pretrained embedding model: it can happen with proper names, spelling errors, or even rare words like 'rattan'. In general the solution is to simply discard the words which are not in the pretrained model, instead of raising an error. I can't answer your other question, I'm not knowledgeable enough about BERT and image captioning (fyi it's a quite advanced task, it might be a bit ambitious). $\endgroup$
    – Erwan
    Jan 20, 2022 at 15:52
  • $\begingroup$ Thanks a lot for your answer and help and your time :-) $\endgroup$
    – Begnnier
    Jan 21, 2022 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.