Skip to main content
5 of 6
Rollback to Revision 3 - Edit approval overridden by post owner or moderator
Stephen Rauch
  • 1.8k
  • 11
  • 22
  • 34

k-means clustering data with large number of meaningless values

I am looking to perform k-means on my dataset which contains a large number of 0 values.

Edit: the last value you see is different to the others, that is simply the sum of transactions, not related to the categorical frequency count.

Example:

[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 189200.579626]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.06556796]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.46e-06]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.46e-06]

Each feature is a frequency count of how many times the associated id is seen within a category.

When I run k-means, I see that most of the data is clustered in one cluster.

+------------+--------+----------------------+
| cluster_id |  size  | sum_squared_distance |
+------------+--------+----------------------+
|     0      | 659187 |    0.999997057952    |
|     1      |   3    |    1.33333326876     |
|     2      |   1    |         0.0          |
|     3      |   3    |    0.666666716337    |
|     4      |   1    |         0.0          |
|     5      |   1    |         0.0          |
|     6      |   1    |         0.0          |
|     7      |   1    |         0.0          |
|     8      |   1    |         0.0          |
|     9      |   11   |    2.72727286816     |
+------------+--------+----------------------+

I am assuming that this is because the majority of the dataset has not been seen in a feature category and therefore has a value of 0.

What is the best way to overcome this, drop rows where a 0 is seen across each category? Are these rows meaningless to clustering?

New to k-means. Thanks,

moved from: https://stackoverflow.com/questions/39130303/k-means-clustering-data-with-large-number-of-meaningless-values