K Means giving poor results

Question

I have several user names and their salaries. Now I need to cluster user based on their salaries. I am using KMeans clustering and following is my code

from sklearn.cluster import KMeans
from sklearn.preprocessing import LabelEncoder
import pandas as pd

le = LabelEncoder()
data = pd.read_csv('kmeans.data',header=None, names =['user', 'salary'])

# Numerical conversion
data['user'] = le.fit_transform(data['user'])
km = KMeans(n_clusters=4, random_state= 10, n_init=10, max_iter=500)
km.fit(data)

data['labels'] = le.inverse_transform(data['user'])
data['cluster'] = km.labels_

print data

But my results are bad and there are lot of overlapping salaries.

Is there anything wrong in the code ? How to improve the results ?

Or whether clustering is not a right approach here ? Then how can I cluster users only based on salary ?

 km.fit(data['salary'])

EDIT:

I figured out a way to solve my problem using numpy.reshape

km.fit(data['salary'].reshape(-1,1))

Answer 1

K-means is based on the assumption that the data is "translation invariant" (more precisely: variance does, and k-means is variance minimization).

In other words, it assumes that a difference of d=(x-y)^2 is of the same importance everywhere. Because of this, k-means does not work on skewed data. Furthermore, because of the square, it is sensitive to outliers and other extreme values.

For salaries and other monetary values, this usually does not hold. The difference between \$0 and \$1000 is massive, and not the same as a salary difference of \$100000 to \$101000. Salaries are usually rather skewed, and you often have some extreme values.

Converting the "user" attribute to a numerical value is outright statistical nonsense. What's variance worth in this attribute? K-means is for continuous numerical data only, and converting data does not chnage the nature, only the encoding - it's still inappropriate.

Answer 2

This is not a 'clustering' problem as much at is it an 'interval' problem since you only have 1 dimension.

You can use an iterative process like Jenk's natural break optimization in order to figure out how large to make your intervals.

As other posters have said, do not user names as a clustering dimension unless you really think that variations in letters of a name are meaningful in some way (do you really think all the Dan's are paid similarly?).

Answer 3

I think the problem here is using the name as a dimension. You can, but you have to use a more robust distance metric between names (string). As far as I know, LabelEncoder just assign an int considering the element's order of ocurrence in a unique list. You could try a different hashing (string to int) or Levenshtein_distance as a distance metric