Clustering of numerical data

Question

I am trying to do clustering in my dataset which has 4 numerical fields. Please find the file attached : http://www.filedropper.com/example_3

I tried with this code:

from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=2, random_state=0, max_iter = 300).fit(dffinal)

I know that there are 2 classes in this example, that was the reason i tried with 2 clusters. Out of 4200 rows, first 3196 rows belong to a class and the remaining rows belong to an another class.

But when i do clustering, cluster labels are randomly assigned, and accuracy is less than 10%. Just wondering whether my features are not good enough for clustering or should I try with some other clustering algorithm.

Any help would be appreciated. Thanks.

Answer 1

You forgot to preprocess your data.

K-means is really sensitive to scale and outliers.

Also: clustering is not classification.

It may well be that e.g. one of your classes has a dense subtype. Just because the clustering found something else than your labels (and it's not supposed to find the same, is it?) does not mean it failed. It just didn't classify the data the same way as your labels (but if that is what you wsnt, then you should have been using a classification method instead.)

Answer 2

I suspect that clustering this data will not be very productive. Just make a simple plot showing two variables at a time. None of these seems to give evidence of natural clustering. One of these plots may give you a glimmer of hope: Count vs Duration3 There is something going on here that some points have Count bigger than 10, others have Duration3 bigger than 10^6 and those two things never happen together. This might hint at a mixture of two groups, but I doubt that clustering is the way to get at that. Most of the points are in the puddle near the origin.

Answer 3

How about this? Convert two fields in a dataframe to an array, and feed that into your kmeans algo to start generating centroids.

#format the data as a numpy array to feed into the K-Means algorithm
data = np.asarray([np.asarray(df['Field1']),np.asarray(df['Field2'])]).T

# computing K-Means with K = 5 (5 clusters)
centroids,_ = kmeans(data,5)
# assign each sample to a cluster
idx,_ = vq(data,centroids)

Finally, map cluster numbers back to IDs of the data set you are working with.

details = [(name,cluster) for name, cluster in zip(returns.index,idx)]

for detail in details:
    print(detail)

That's basically it. See the link below for all details related to what I described above.

https://www.pythonforfinance.net/2018/02/08/stock-clusters-using-k-means-algorithm-in-python/