Couple PCA plot and clusters to labels

Question

I am trying my first 'project' concerning machine learning and I am a bit stuck. However, I am not sure if it's even possible but here goes my question.

What I want to achieve is clustering user groups based on the amount of visits a user does on a certain website. So I started out with this feature matrix:

USER    abc.be  abc.be/a    abc.be/b    xyz.be  xyz.be/a
123      0        0           0            0      1
456      1        0           1            0      0
789      2        3           1            0      0
321      1        0           1            0      1
654      1        1           1            1      1
987      0        1           0            3      0

So I got in this example 5 features (my 5 different websites). So then I used PCA to come to 2 dimensions, so I could plot it and see how it went.

My feature matrix (in my example) is 5 columns * 6 rows.

My PCA matrix is 2 columns * 6 rows.

I came to this plot (please note that this plot uses different data then the example but the idea is the same)

The green points are my PCA points The red circles are my K-Means centroids.

But the part I am struggling with is this: so I got my clusters (red circles) but how can I use this to say:"Looks like most users go to either site A or site B)?

So how can I couple my clusters to a feature label from my feature matrix?

Or how does one approach this?

Any help is appreciated :)

Answer 1

So how can I couple my clusters to a feature label from my feature matrix?

Principal Components are not intuitive features. What seems common here is to cluster users based on PCs and investigate clusters based on original features afterward i.e. extract different clusters and plot data based on different subsets of features and use different colors for different clusters. It might give you some intution.

These two paths can be seen in many PCA results where the information varies with e.g time.

For instance in your case a beginner user visits less than an old one and the number of new users are most probably higher than old ones so data will be more dense around origin and far it gets from the origin less dense it becomes. Such a phenomenon affects your PCs as well so you'll see some paths along different lines in PC space.

Hope it helps :)

Answer 2

I am not sure about what you are trying to do. But generally speaking, you can look at this paper for the relation between k-means clustering and PCA, especially Theorem 3.3.

Answer 3

I think you are trying to see which original (not PCA) features contribute to which cluster users fall into, yes?

First, be aware that you just reduced the dimensionality of a 5d dataset down to 2d. You need to be careful of how much variance of the data you just threw away by projecting it into 2d. You can easily calculate this. If your 2d features account for 95% of the variance of the data, then great! You've got some valid insights about which features are important. If lower, like say 40%, then not so much.

(From a qualitative standpoint, just plot the clusters with different colors. That will give you some idea of how much variance you threw away - if the clusters look contiguous and not a lot of mixing, then qualitatively, you didn't throw away too much variance.)

Second, realize that in order to intuit on which original feature values contribute to which clusters, you'll need to use the original features.

The real answer to your question is that after that you should 1) cluster in 5d, 2) use a classifier, and 3) see the important features using 5d feature vectors. This is critical because in 2d, just because you might have kept most of the variance doesn't mean that last bit might have contained very discriminatory (read: important) information that might improve your underlying clustering error and give you a much more accurate answer. Most importantly, the feature importance will be computed on the actual features, not the PCA linear-combination features which don't translate directly to single site visit histories.

Hope that helps.

Answer 4

Take the cluster centers and project them back to the original feature vector using the inverse transform of your PCA. That gives you an intuition of what the clusters represent. As a second step you can take all original feature vectors that were assigned to one cluster and calculate histograms on each feature dimension. This will give you a feeling of in-cluster variation and cluster boundaries along the original feature dimensions.