2
$\begingroup$

I am working on a clustering problem which has 13000 observations and 15 features. Around 3000 observations in the dataset has zero in every features ( i.e all values zero in 3000 rows). I am trying to do clustering on top of it. What is a better way to deal with it ? I have few things in my mind but would like to get clarity on :

  1. Check for number of rows with all zero and remove them ?
  2. Include the rows with zero value in every feature and let the clustering algorithm handle the same?

Also,please suggest if there are any better way to handle the same.

Note : I am using k-means clustering.

$\endgroup$
3
$\begingroup$
  1. If you expect that all zeros is a result of error in the measuring of the features (i.e. the observations should not be all 0s but they are), then I would say: Keep all the data, but increase k (from k-means) by 1. This extra one will hopefully become the class of all these wrong observations.

  2. If you expect that all zeros is correct (i.e. these observations are indeed all zeros) just keep them and go on as normal.

Note: If you keep using K-Means having all these extra observations should be fine. However, if you switch to another algorithm which takes densities into account (e.g. Mean Shift) then having all these extra observations might influence your model in ways that you do not expect. For example, the observations [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.1] might end up being far from the all 0s cluster.

|improve this answer|||||
$\endgroup$
3
$\begingroup$

It's a matter of data quality so it depends how the dataset was built:

  • Either these instances are meaningful, i.e. it makes sense that an observation would have zeros for all the features and that it would happen that often.
  • Or these are the result of an error, typically the complete absence of measurement for these observations.

Naturally one wants to keep the instances in the data in the former case but (usually) not in the later case, because the values don't represent an actual data point so they would introduce a massive bias for the clustering algorithm.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.