Concerning the notion of word embeddings, Skip-Gram methods aim for computing the probability of a word given its neighborhood. I do not understand the rationale behind it, since it is possible to infer this information by looking directly into the co-occurrence matrix.

In general, I cannot understand those methods aiming to capture as much relevant information from the original co-occurrence matrix as possible. Isn't it easier to work on the co-occurrence matrix directly?

Thank you very much in advance.

  • $\begingroup$ Skip-grams methods are about increasing the size of the context window without increasing the order, which exponentially increases the amount of data you have to deal with. It's a clever hack. $\endgroup$ – Emre Aug 30 '17 at 14:58
  • $\begingroup$ By order I refer to the number $n$ in an $n$-gram. $\endgroup$ – Emre Aug 30 '17 at 15:57

Basically, skip-gram aim for computing the probability of a context given a word (it is CBOW that does the opposite). BUT, it does not simply learn a co-occurrence matrix, it compresses the information in a low dimensional space (for example 300 for a 100000 original dimension). By doing this, it learns continuous low dimensional representation for the words.

It is proved that word2vec actually factorizes a word-context pointwise mutual information matrix (close to the co-occurrence matrix). I think this article might give you a better understanding of the underlying processes.

  • $\begingroup$ Thank you very much for your interesting contribution! Yes, but why we need a low dimensional representation? I think that current hardware has no problem for dealing with large matrices. $\endgroup$ – Jorgemar Aug 30 '17 at 14:53
  • 2
    $\begingroup$ Compression (low dimension) is the key to learn invariance. You should learn about PCA and LSA to understand why we go from high dimensions to low dimensions. It is not always a question of computational power. en.wikipedia.org/wiki/Latent_semantic_analysis $\endgroup$ – Robin Aug 30 '17 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.