How is CBOW different from building PMI matrix and then reducing using PCA

Question

PMI matrix and reduction using PCA: Based on the number of times 2 words appear together (in a certain pre-defined window), and the individual frequency of words, we build the PMI matrix. Then reduce it using PCA, to get dense representations of each word in the corpus, which are able to capture some semantics of the text

CBOW: Learning word representations through a neural network, whose end objective is to maximize the probability of correct word pairs. The probability values are known in advance by counting the number of times a word is appearing in another word's context in the training data.

Both of these methods are using counts, and then getting a dense word representation. Is there a definitive advantage of one over the other? Why was CBOW introduced at all when the former method is doing the exact same job?

Answer 1

There are several differences:

• CBOW is not equivalent to matrix factorisation of a PMI matrix, it is SkipGram whose loss function is minimised when $W^\top C=PMI$ (see Goldberg & Levy paper, 2014)
• SkipGram is not equivalent to PCA of the PMI matrix since the loss function is not the least squares loss (again see Goldberg & Levy, 2014 or "What the vec? ... " Allen et al, 2019)

It was not known that SkipGram has anything to do with PMI when the algorithm was first proposed (Mikolov et al, 2014). Having shown the connection, Goldberg & Levy show comparisons of W2V embeddings and PCA of PMI etc and they give broadly comparable performance on some tasks.