# Extracting useful features for k-means clustering

So say suppose I have a data-set with features being either present or not i.e. 0or 1. Now I want to identify the features which really help in the clustering.

Like say I have 4 training examples. Now say I have a feature which is present i.e = 1 in all the training example, thus I can remove the feature as it dies not help me. Now let's talk about 2 more features, if the number of training examples they are present in common is high, they also do not help much in clustering (think of 2 highly overlapping circles in a Venn Diagram). So in this way I want to find features which has a significant impact on the clustering i.e mostly non-overlapping features.

Is there any good way to do this? (my features are all represented in binary, either it is there or it is not).

There are many ways for features selection:

One possible way (Wrapper methods), is to start by clustering with all related features and save the clusters, then cluster again after removing 1 feature and compare the obtained clusters (without the feature) to the clusters (with the feature). If the difference between the results are negligible then you can eliminate that feature.

Filter methods include what you mentioned in your question, where you remove the feature that changes rarely or doesn't change at all which can be expressed mathematically by "entropy", "information gain", "Chi square test"...etc. these concepts measure the impurity of a feature.

To sum up, "Wrapper" methods usually provide better results but they are computationally more expensive.

For more please see this

• I was kind of thinking the same thing...but for large datasets it can be quite resource consuming – DuttaA Jun 14 '18 at 10:34

Well, for binary data, k-means does not make that much sense.

Assuming that your 4 training examples are in different clusters, you should probably scale all attributes to have the same variance in these four instances. Except that 4 is too little to do a meaningful variance estimation... Then use the four training examples as initial centroids.

• Why do you think k-means is not useful for binary data? – DuttaA Jun 15 '18 at 6:09
• Because it the means are not binary anymore, and the idea of least-squares in k-means relates to assuming Gaussian errors masking k signals. There is no Gaussian error in binary data. – Has QUIT--Anony-Mousse Jun 15 '18 at 6:17