1
$\begingroup$

I'm a new DL4J user, and I'm running all the works of Shakespeare through a Word2Vec neural net. I've got a pretty basic question about how to understand the results so far. In the below example, there's an obvious association with the "ing" in king and the "ing" in other words that probably don't have much to do with king. Am I missing something about how a word2vec formula uses the characters inside the words it is mapping? Or is my net just really untrained?

Also, what does the cosine distance between those example words say to you about the results, if anything? Thank you for your advice!

   List<String> abc = vec.similarWordsInVocabTo("king", 0.8); //80% similar
   System.out.println(abc);

   String[] words = {"woman", "king", "boy", "child", "human"};
   for (String word : words) {
       System.out.println(vec.similarity("man", word));
   }

Output - Similar words to king:

[taking, drinking, kingly, picking, waking, singing, wringing, knight, feigning, beginning, ink, thinking, kin, knocking, making, bringing, knowing, lingring, winking, neighing, king-, kings, asking, stinking, king, liking]

Output - Vector similarity between "man" and woman, king, boy, child, human:

woman:   0.8305895924568176
king:    0.00203840178437531
boy:     0.2974374294281006
child:   0.4752597510814667
human:  -0.10414568334817886
$\endgroup$

1 Answer 1

0
$\begingroup$

Word2Vec algorithm does not go inside words. Word “king” is never used as a gerund, so there is no reason why it should be similar to gerunds.

My guesses are:

  • Your corpus might by wrongly tokenized. Maybe there are some OCR-related errors with word splitting something like “li-↲ king”.

  • You might be using a different algorithm for getting the embeddings (e.g., FastText) that goes inside the words and infers the word embedding as some embeddings of character n-grams the word consists of.

On the other hand, words similar to man look fine. If you think about how Word2Vec (and also FastText) is trained, you should not ask a question: ”Do the words have as similar meaning as possible?” but rather “Does the word appear similarly frequently in a similar context Shakespear's works?”

(Of course, when the embeddings are trained on data which is large enough, there is almost no difference between these two questions.)

$\endgroup$
1
  • $\begingroup$ Thank you for your feedback! That's a good point about a word being used as a gerund, but you're right, some of the words seem just fine as far as the proper relation mapping. As a follow up, I tested "shine" in the same data set and got strong relations to words like "shone, lines, whiteness, shining, sunshine" BUT also to "swine, shrine, mines, and sheep-sheering (what!?)". I realized that one of the ways it may be analyzing distance is character-based in order to capture different tenses. I have no evidence of that, it just makes sense why it could be built that way. $\endgroup$
    – DPCII
    Jan 28, 2020 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.