I am looking to perform k-means on my dataset which contains a large number of 0 values.
Edit: the last value you see is different to the others, that is simply the sum of transactions, not related to the categorical frequency count.
Example:
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 189200.579626]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.06556796]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.46e-06]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.46e-06]
Each feature is a frequency count of how many times the associated id is seen within a category.
When I run k-means, I see that most of the data is clustered in one cluster.
+------------+--------+----------------------+
| cluster_id | size | sum_squared_distance |
+------------+--------+----------------------+
| 0 | 659187 | 0.999997057952 |
| 1 | 3 | 1.33333326876 |
| 2 | 1 | 0.0 |
| 3 | 3 | 0.666666716337 |
| 4 | 1 | 0.0 |
| 5 | 1 | 0.0 |
| 6 | 1 | 0.0 |
| 7 | 1 | 0.0 |
| 8 | 1 | 0.0 |
| 9 | 11 | 2.72727286816 |
+------------+--------+----------------------+
I am assuming that this is because the majority of the dataset has not been seen in a feature category and therefore has a value of 0.
What is the best way to overcome this, drop rows where a 0
is seen across each category? Are these rows meaningless to clustering?
New to k-means. Thanks,
moved from: https://stackoverflow.com/questions/39130303/k-means-clustering-data-with-large-number-of-meaningless-values