# What are the advantages of Skip-Gram methods?

Concerning the notion of word embeddings, Skip-Gram methods aim for computing the probability of a word given its neighborhood. I do not understand the rationale behind it, since it is possible to infer this information by looking directly into the co-occurrence matrix.

In general, I cannot understand those methods aiming to capture as much relevant information from the original co-occurrence matrix as possible. Isn't it easier to work on the co-occurrence matrix directly?

Thank you very much in advance.

• Skip-grams methods are about increasing the size of the context window without increasing the order, which exponentially increases the amount of data you have to deal with. It's a clever hack.
– Emre
Aug 30, 2017 at 14:58
• By order I refer to the number $n$ in an $n$-gram.
– Emre
Aug 30, 2017 at 15:57

## 1 Answer

Basically, skip-gram aim for computing the probability of a context given a word (it is CBOW that does the opposite). BUT, it does not simply learn a co-occurrence matrix, it compresses the information in a low dimensional space (for example 300 for a 100000 original dimension). By doing this, it learns continuous low dimensional representation for the words.

It is proved that word2vec actually factorizes a word-context pointwise mutual information matrix (close to the co-occurrence matrix). I think this article might give you a better understanding of the underlying processes.

• Thank you very much for your interesting contribution! Yes, but why we need a low dimensional representation? I think that current hardware has no problem for dealing with large matrices. Aug 30, 2017 at 14:53
• Compression (low dimension) is the key to learn invariance. You should learn about PCA and LSA to understand why we go from high dimensions to low dimensions. It is not always a question of computational power. en.wikipedia.org/wiki/Latent_semantic_analysis Aug 30, 2017 at 15:04