Dimension reduction of Word Embeddings: PCA vs. TSNE

Question

I am pretty new to DS. I have a general question regarding the limitations of visualizing word embeddings using PCA.

I've learned so far that when using PCA (e.g. with sklearn), the explained variance (explained_variance_ratio_) describes how much of the variance is explained by each principal component. For example, once 80% of the variance is explained (or elbow), one can be fairly confident that this number of PCs is a good approximation to describe the variance of the data in low-dimensional space. In the case of visualisation, one is limited to 2D or 3D for obvious reasons.

Applying PCA to a word embedding matrix (300 columns for vectors, N rows for N words), I have a small contribution from the first two PCs (PC1: 4%, PC2: 3.5%). I have a linear decay up to PC_N.

I am confused now, PCA describes so little information for the first two or three components. In many tutorials, PCA is directly applied to only two/three dimensions without checking the contribution of the PCs. Is this the reason why some people use T-SNE from the very beginning?

Answer 1

First of all, though both PCA and t-SNE are dimension reduction methods, they are designed to serve different purposes. PCA seeks a low-rank representation in a sense (maximizes the variance in projection space), while t-SNE produces a low-dimension representation specially suited for visualization purpose.

Word embedding matrix is already a low-rank representation, so there is not much juice left to squeeze for PCA (trying to get a even-lower-rank matrix of a low-rank matrix). This is why you see a linear decay (instead of power decay typically) of component coefficients. On the other hand, a low-rank matrix may still have much to gain from t-SNE if we want to visually inspect it.

Again, PCA and t-SNE are just tools - which one to pick is dependent on what the purpose is.