Differences between applying KMeans over PCA and applying PCA over KMeans

Question

Short question:

As stated in the title, I'm interested in the differences between applying KMeans over PCA-ed vectors and applying PCA over KMean-ed vectors.

Long question:

Let's suppose we have a word embeddings dataset. Each word in the dataset is embeded in R300.

We want to perform an exploratory analysis of the dataset and for that we decide to apply KMeans, in order to group the words in 10 clusters (number of clusters arbitrarily chosen).

After doing the process, we want to visualize the results in R3. We could tackle this problem with two strategies;

Strategy 1 - Perform KMeans over R300 vectors and PCA until R3:

1. Apply KMeans to the R300 embeddings.
2. Perform PCA to the R300 embeddings and get R3 vectors.
3. Plot the R3 vectors according to the clusters obtained via KMeans

Result: http://kmeanspca.000webhostapp.com/KMeans_PCA_R3.html

Strategy 2 - Perform PCA over R300 until R3 and then KMeans:

1. Perform PCA to the R300 embeddings and get R3 vectors.
2. Apply KMeans to the R3 embeddings.
3. Plot the R3 vectors according to the clusters obtained via KMeans

Result: http://kmeanspca.000webhostapp.com/PCA_KMeans_R3.html

Are there any differences in the obtained results? Any interpretation?

In case both strategies are in fact the same. Why is that?

Answer 1

There is a difference. In your first strategy, the projection to the 3-dimensional space does not ensure that the clusters are not overlapping (whereas it does if you perform the projection first).

This is because some clusters are separate, but their separation surface is somehow orthogonal (or close to be) to the PCA.

If you increase the number of PCA, or decrease the number of clusters, the differences between both approaches should probably become negligible.

Answer 2

I would recommend applying GloVe info available here: Stanford Uni Glove to your word structures before modelling.

This way you can extract meaningful probability densities. If you then PCA to reduce dimensions at least you have interrelated context that explains interaction.

Effectively you will have better results as the dense vectors are more representative in terms of correlation and their relationship with each other words is determined. This is due to the dense vector being a represented form of interaction. This process will allow you to reduce dimensions with a pca in a meaningful way ;)