I am pretty new to DS. I have a general question regarding the limitations of visualizing word embeddings using PCA.
I've learned so far that when using PCA (e.g. with
sklearn), the explained variance (
explained_variance_ratio_) describes how much of the variance is explained by each principal component. For example, once 80% of the variance is explained (or elbow), one can be fairly confident that this number of PCs is a good approximation to describe the variance of the data in low-dimensional space. In the case of visualisation, one is limited to 2D or 3D for obvious reasons.
Applying PCA to a word embedding matrix (300 columns for vectors, N rows for N words), I have a small contribution from the first two PCs (PC1: 4%, PC2: 3.5%). I have a linear decay up to PC_N.
I am confused now, PCA describes so little information for the first two or three components. In many tutorials, PCA is directly applied to only two/three dimensions without checking the contribution of the PCs. Is this the reason why some people use T-SNE from the very beginning?