0
$\begingroup$

I am pretty new to DS. I have a general question regarding the limitations of visualizing word embeddings using PCA.

I've learned so far that when using PCA (e.g. with sklearn), the explained variance (explained_variance_ratio_) describes how much of the variance is explained by each principal component. For example, once 80% of the variance is explained (or elbow), one can be fairly confident that this number of PCs is a good approximation to describe the variance of the data in low-dimensional space. In the case of visualisation, one is limited to 2D or 3D for obvious reasons.

Applying PCA to a word embedding matrix (300 columns for vectors, N rows for N words), I have a small contribution from the first two PCs (PC1: 4%, PC2: 3.5%). I have a linear decay up to PC_N.

I am confused now, PCA describes so little information for the first two or three components. In many tutorials, PCA is directly applied to only two/three dimensions without checking the contribution of the PCs. Is this the reason why some people use T-SNE from the very beginning?

$\endgroup$
1
  • $\begingroup$ Online tutorials tend to have a bias toward simplicity rather than toward completeness. Also, you'd better use U-MAP instead of t-SNE. $\endgroup$
    – noe
    May 3, 2023 at 15:52

1 Answer 1

1
$\begingroup$

First of all, though both PCA and t-SNE are dimension reduction methods, they are designed to serve different purposes. PCA seeks a low-rank representation in a sense (maximizes the variance in projection space), while t-SNE produces a low-dimension representation specially suited for visualization purpose.

Word embedding matrix is already a low-rank representation, so there is not much juice left to squeeze for PCA (trying to get a even-lower-rank matrix of a low-rank matrix). This is why you see a linear decay (instead of power decay typically) of component coefficients. On the other hand, a low-rank matrix may still have much to gain from t-SNE if we want to visually inspect it.

Again, PCA and t-SNE are just tools - which one to pick is dependent on what the purpose is.

$\endgroup$
1
  • $\begingroup$ Thx for your answer $\endgroup$
    – Bernardo
    May 11, 2023 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.