# How is PCA is different from SubSpace clustering and how do we extract variables responsible for the first PCA component?

New update:

I understand PCA components ensure we select variables responsible for high variance, but I would like to know how to extract key variables responsible only for high variance through PCA components.

Ideally, a simple example would help.

This is my code:

#Implementing PCA for visualizing after Kmeans clustering

# Interpret 3 cluster solution
model3=KMeans(n_clusters=3)
model3.fit(clus_train)
clusassign=model3.predict(clus_train)
# plot clusters


'''The new variables, called canonical variables, are ordered in terms of the proportion of variance and the clustering variables that is accounted for by each of the canonical variables. So the first canonical variable will count for the largest proportion of the variance. The second canonical variable will account for the next largest proportion of variance, and so on. Usually, the majority of the variance in the clustering variables will be accounted for by the first couple of canonical variables and those are the variables that we can plot. '''

from sklearn.decomposition import PCA
pca_2 = PCA(2) # Selecting 2 components
plot_columns = pca_2.fit_transform(clus_train)
plt.scatter(x=plot_columns[:,0], y=plot_columns[:,1], c=model3.labels_,)


Observations are more spread out indicating less correlation among the observations and higher within cluster variance.

plt.xlabel('Canonical variable 1')
plt.ylabel('Canonical variable 2')
plt.title('Scatterplot of Canonical Variables for 3 Clusters')
plt.show()

• Many advantages, like collapsing correlated variables... Apr 3, 2017 at 9:11
• Thanks Student for insights. Can you give an example illustration where I can undertstand what you mean... Because After getting PCA components, I used it only for visualizing. I would like to know how this can be used for collasing correlated variables? Actually I was using correlation function before clustering itself, So your suggestion would eliminate that redundency in my code Apr 3, 2017 at 9:19
• Please, learn the markdown format (use the help button) to make your question readable... Apr 4, 2017 at 18:20

Reducing the dimensionality of a dataset with PCA does not only benefits humans trying to look at the data in a graspable number of dimensions. It is also useful for machine learning algorithms to be trained on a subset of dimensions. Both to reduce the complexity of the data and the computational cost of training such machine learning model.

• Thanks Tony for insights. Can you give an example illustration where I can undertstand what you mean... Because After getting PCA components, I used it only for visualizing. I would like to know how this can be used for reducing complexity ? Apr 3, 2017 at 9:17
• Check this out: jotterbach.github.io/2016/03/24/Principal_Component_Analysis and scikit-learn.org/stable/modules/… for some examples
– tony
Apr 3, 2017 at 10:53
• Thanks Tony for an understandable example. I tried to execute... tl.set_color(plot1[0].get_color()) seems to give an error NameError: name 'plot1' is not defined I am using python 3.5 on spyder Apr 3, 2017 at 13:11

PCA is a very common technique, so you might want to google around. PCA is awfully common for data visualisation, but it has many other uses.

For instance, if you want to fit a linear regression on average income. Now, you have collected 500+ predictors, but lots of them are correlated like:

• How much the person pays tax last year
• How much the person pays tax the year before
• How much the person pays tax three years ago
• ....

Those predictors are highly correlated and might present modelling problems in your linear model. A very common technique is to use PCA to reduce into a set of reduced orthogonal principal components. You can then use those components for building your model.

https://stats.stackexchange.com/questions/22665/how-to-use-principal-components-as-predictors-in-glm