0
$\begingroup$

I am working with a data set that, besides customer age and income, tells the balance a customer has in different type of bank accounts: Checking, Shares, Investment, Savings, Deposit, Mortgage, Loan, and Certificates. For accounts other than Checking, 0 represents that the account does not exist for the customers. There are 9800 customer observations with roughly 6000 checking accounts and 4000 savings accounts. For the others, the observations are less than 300.

I have to use K-Means Clustering analysis for the segmentation with the objective to understand how customers use savings and investment offerings and I am using the Elbow Method to predict the number of clusters. I am confused whether to use a variable like Investment with just 250 observations with another like Savings that has 4000. If I do use such variables, then these are heavily positively skewed and I'm not sure if K-Means handles that well. Can someone advise whether to include such variables or not?

$\endgroup$
  • $\begingroup$ I strongly doubt that the k-means algorithm will yield any useful results here, and I dare to claim that you'll have a hard time to argue why the result is "correct" statistically. It is the wrong data kind for this algorithm. K-means is appropriate if you have k well-separated signals + Gaussian errors, and clearly your data does not have Gaussian errors. $\endgroup$ – Anony-Mousse Nov 16 at 16:34
  • $\begingroup$ From whatever I have researched over the past week, I can only concur with you. However, since this is an academic exercise, my hands are tied and I just have to create the best possible scenario out of it using K-Means. Until now, I am inclined to avoid variables / bank accounts that have low representation. $\endgroup$ – Obaid Khan Nov 16 at 16:43
  • $\begingroup$ The point of academia is do things the "correct" and best way. I'd agree with you if it's "because the customer demands k-means". But in academia, providing a better approach than asked for should always be acceptable. Also, the dreadful "elbow" method most likely will not find a sound elbow at all, but just the usual curve on random noise. So you should also point out that there is no "optimal" k, and the elbow method suggests that KMeans failed. $\endgroup$ – Anony-Mousse Nov 16 at 17:51
  • $\begingroup$ Thank you and I would definitely suggest an alternate approach in my final report and presentation but for the sake of this exercise, can you suggest an approach that is "less worst"? Would the results further degrade if such variables are included? $\endgroup$ – Obaid Khan Nov 17 at 0:27
0
$\begingroup$

I suggest you use an algorithm that accommodates categorical variable since there are missing data. You can one hot encode it so missing data will be relevant. Making it zero will be misleading.

Try algorithms like tSNE and Self Organizing Map and use the Jaccard/Tanimoto distance.

$\endgroup$
  • $\begingroup$ Thank you for your reply. Unfortunately, I do not have a lot of room to play with here in terms of the approach. The dynamics of the task are such that I can only use K-Mean algorithm. I am not sure if we can call those variables categorical considering that the data is continuous for account having balance greater than 0. Do you think I should avoid such variables altogether and use Checking and Savings accounts which have at least 50 percent observations? $\endgroup$ – Obaid Khan Nov 16 at 14:02
  • $\begingroup$ The problem is the missing observation since making it zero doesn't make sense at first glance. But still try it and see if it make sense. Try different approaches so you will know what make sense. In tanimoto distance, you can have mixed categorical and continuous variable. $\endgroup$ – bonez001 Nov 16 at 14:06
  • $\begingroup$ In general, avoiding variables just give different clusters. Just try all cases that you think. You will not lose anything. In general you will never know what those variables will give. They might be informative in the end. $\endgroup$ – bonez001 Nov 16 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.