Extracting useful features for k-means clustering

Question

So say suppose I have a data-set with features being either present or not i.e. 0or 1. Now I want to identify the features which really help in the clustering.

Like say I have 4 training examples. Now say I have a feature which is present i.e = 1 in all the training example, thus I can remove the feature as it dies not help me. Now let's talk about 2 more features, if the number of training examples they are present in common is high, they also do not help much in clustering (think of 2 highly overlapping circles in a Venn Diagram). So in this way I want to find features which has a significant impact on the clustering i.e mostly non-overlapping features.

Is there any good way to do this? (my features are all represented in binary, either it is there or it is not).

Answer 1

There are many ways for features selection:

One possible way (Wrapper methods), is to start by clustering with all related features and save the clusters, then cluster again after removing 1 feature and compare the obtained clusters (without the feature) to the clusters (with the feature). If the difference between the results are negligible then you can eliminate that feature.

Filter methods include what you mentioned in your question, where you remove the feature that changes rarely or doesn't change at all which can be expressed mathematically by "entropy", "information gain", "Chi square test"...etc. these concepts measure the impurity of a feature.

To sum up, "Wrapper" methods usually provide better results but they are computationally more expensive.

For more please see this

Answer 2

Well, for binary data, k-means does not make that much sense.

Assuming that your 4 training examples are in different clusters, you should probably scale all attributes to have the same variance in these four instances. Except that 4 is too little to do a meaningful variance estimation... Then use the four training examples as initial centroids.