Timeline for Gensim doc2vec error: KeyError: "word 'senseless' not in vocabulary"
Current License: CC BY-SA 4.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 19, 2023 at 13:47 | comment | added | Darren Cook |
@AnkitRohilla It would be very normal for word_tokenize() to be splitting 'senseless' into two tokens, which would explain all the behaviour you see. (I.e. why else would you be using a tokenize function!) But without more information about the modules you imported, or a fully reproducible example (which always shows all the import commands), or more information, we are reduced to guessing.
|
|
| Jan 19, 2023 at 11:05 | comment | added | Ankit Rohilla | I think if model_dbow1.most_similar('new') is working, then model_dbow1.most_similar('senseless') should also work. I checked "senseless" word is not in the model_dbow1.wv.vocab after training. I checked "trim_rule" parameter of doc2vec. I think vocab is being trimmed. | |
| Jan 17, 2023 at 9:13 | comment | added | Darren Cook |
What did my suggestions do? BTW, looking at your updated question, what is word_tokenize()? If you run most_similar( word_tokenize('senseless') ) does it work?
|
|
| Jan 17, 2023 at 6:31 | comment | added | Ankit Rohilla | For other word like 'new', it's working | |
| Jan 16, 2023 at 21:04 | history | answered | Darren Cook | CC BY-SA 4.0 |