4
$\begingroup$

I am using word2vec to train a big corpus of textual information. While analyzing the scatter plot of words I found that it has not only grouped numbers together but also ordered them based on their value. I am able to understand that it groups together words sharing similar context but not able to understand how it got the values also correctly. Can someone please give an intuitive explanation of how this happens? I am attaching the image where I have zoomed in on the numbers.word2vec zoomed-in output

$\endgroup$
  • $\begingroup$ what kind of dataset is it ?. Does it has sentence numbers at the start ?. Did you train it sentence by sentence or as a whole paragraph containing several numbers. $\endgroup$ – chmodsss Dec 13 '16 at 16:35
  • 3
    $\begingroup$ Intuitively I would say that it will be true for small numbers and less true for big numbers (7-9-8 seems to go that way). When we speak we often say things like: "I ate 1 or 2 apples" or "I will buy 2 or 3 things" but rarely "I will buy 3 or 2 things". I think it's how it gets to order the numbers. Bigger numbers will appear less in a corpus so they will most probably not be ordered as precisely. $\endgroup$ – Eskapp Dec 13 '16 at 21:13
  • 1
    $\begingroup$ Also possibly related: en.wikipedia.org/wiki/Benford's_law - it may apply if you have a large enough corpus and word2vec has encoded something to do with word rarity. Just a guess though. $\endgroup$ – Neil Slater Dec 13 '16 at 22:36
  • $\begingroup$ @NeilSlater Benford's law talks about the first digit of big numbers. But does word2vec breaks up numbers into digits? My thought was that it treats numbers as words and doesn't break up into digits. Please clarify. $\endgroup$ – Shamy Dec 15 '16 at 2:15
  • $\begingroup$ @shamy I think you are right (although I don't know word2vec), but there could be similar ratios between single digit numbers taken in a large enough corpus. The only semantic clustering I can really think of though is 1 vs other numbers due to use of singular vs plural terms in rest of sentence. Hence I cannot make an answer. $\endgroup$ – Neil Slater Dec 15 '16 at 7:40
1
$\begingroup$

Attempting an answer from the understanding I have of word2vec and deep learning through my personal experience and some live presentations of Yoshua Bengio and Yann Le Cun that I had the chance to attend.

As mentioned in the question, I think too that numbers are grouped together as they are have similar functions in the sentences where they are found.

For the reason of how they get ordered, as said in my comment: Intuitively I would say that it will be true for small numbers and less true for big numbers (the way 7-9-8 are ordered seems to go that way). When we speak we often say things like: "I ate 1 or 2 apples" or "I will buy 2 or 3 things" but rarely "I will buy 3 or 2 things". I think this is how it ends up ordering the numbers. Bigger numbers will appear less in a corpus so they will most probably not be ordered as precisely.

When looking at tutorials about word2vec, an example that is often given is that, if we see the relationship between 2 words as a vector, the final representation of the words by the model will give that the vector linking "man" to "woman" is approximately the same as the one linking "king" to "queen". Or that the vector linking "Paris" to "France" is approximately equal to the one linking "Rome" to "Italy". This even allows to do operations such as: France + Italy - Paris = Rome! The clear mechanics of how these relationships are so accurate (compared to other approaches) are, as of today, still obscur to the scientific community.

If these kind of operations works for countries and main cities, it seems logical that the vectors "3-2" and "2-1" point in the same direction, i.e. that 1-2-3 are ordered (given that, as said previously, consecutive numbers are more likely to be used together in a sentence). Of course, to achieve this, the numbers have to frequently appear in the training corpus.

I am very happy that someone showed this result with numbers because the way word2vec has been trained in most of the models that are available online did not includ digits. They were replaced by some symbols i.e., 12.34 becoming ##.##, and such observations cannot be made.

Any suggestion to improve this answer is welcome!

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.