14
$\begingroup$

I was working on word2vec gensim model and found it really interesting. I am intersted in finding how a unknown/unseen word when checked with the model will be able to get similar terms from the trained model.

Is this possible? Can word2vec be tweaked for this? Or the training corpus needs to have all the words of which i want to find similarity.

Improve this question
$\endgroup$
1

4 Answers 4

Reset to default
10
$\begingroup$

Every algorithm that deals with text data has a vocabulary. In the case of word2vec, the vocabulary is comprised of all words in the input corpus, or at least those above the minimum-frequency threshold.

Algorithms tend to ignore words that are outside their vocabulary. However there are ways to reframe your problem such that there are essentially no Out-Of-Vocabulary words.

Remember that words are simply "tokens" in word2vec. They could be ngrams or they could be letters. One way to define your vocabulary is to say that every word that occurs at least X times is in your vocabulary. Then the most common "syllables" (ngrams of letters) are added to your vocabulary. Then you add individual letters to your vocabulary.

In this way you can define any word as either

  1. A word in your vocabulary
  2. A set of syllables in your vocabulary
  3. A combined set of letters and syllables in your vocabulary
Improve this answer
$\endgroup$
3
$\begingroup$

The training corpus needs to have all the words of which you want to find similarity.

Improve this answer
$\endgroup$
3
$\begingroup$

word2vec treats words as atoms. To get meaningful vectors for unknown words, you either have to

  • change what these atoms are, e.g. switch to letter n-grams as in jamesmf's answer, or
  • use a different model that explicitly looks at what is inside your words, e.g. the CWE model is easy to use.
Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ github.com/facebookresearch/fastText seems to work well $\endgroup$
    – Joachim Wagner
    Commented Jan 13, 2017 at 16:02
  • $\begingroup$ yea, I tried that but doesn't work well with tasks like morphological segmentation. $\endgroup$
    – gaurus
    Commented Feb 20, 2017 at 13:10
0
$\begingroup$

The word2Vec and FastText fail if the word is not in the vocabulary. Throws an error. It gives a list of score for related words But an unseen word will not be in the vocabulary isn't it? So, how does it solve the unseen word problem?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.