I have a dataset as follows (not the actual data, but representative):

User   Age  Salary  Status_2000  Status_2001  Status_2002  Status_2003  ... Status_2019
John   30   10000   A            A            A            B            ...  A
Mary   25   20000   A            B            B            B            ...  C
Jacob  35   30000   E            F            F            F            ...  A
Jenny  28   22000   A            F            F            A            ...  F

I have over 50,000 rows and I want to perform an analyse the development patterns of the Status from 2000 to 2019. However, the Status are encoded with letters from A to J. I would like to ask whether there are any machine learning methods or clustering methods which could help in this analysis? I initially thought of K-Means but I'm not sure if that is appropriate even with Label Encoding.

In the same dataset, there are also the following columns:

User   Age  Salary  2000_%  2001_%  2002_%  2003_%  ... 2019_%
John   30   10000   0%      0%      20%     25%     ... 100%
Mary   25   20000   0%      10%     40%     60%     ... 80%
Jacob  35   30000   0%      80%     100%    80%     ... 80%
Jenny  28   22000   0%      0%      0%      20%     ... 60%

where the data represents % of the maximum amount from 2000 to 2019, and I want to perform an analysis on the development of the %. Are there any techniques suitable for such analysis?

Thank you all for your help in advance!

  • $\begingroup$ What are you even trying to do? $\endgroup$ – Anony-Mousse Sep 3 at 5:58
  • $\begingroup$ @Anony-Mousse I am trying to see if there are any patterns (i.e. if there are many statuses that take the path A-->A-->B etc) in the data...but not sure what methods to use $\endgroup$ – InvadersMustDie Sep 3 at 7:43
  • 1
    $\begingroup$ Well, that obviously calls for a very different approach than clustering, but rather frequent subsequence mining, doesn't it? But you really need to first clearly identify the problem that you need to solve before trying out random methods. If you don't know what you really want and need, nobody else is. Ans you'll get the usual "use k-means" answers that won't actually solve anything. $\endgroup$ – Anony-Mousse Sep 3 at 9:46
  • $\begingroup$ @Anony-Mousse Ahhh yes, I do have an idea of the problem I want to solve, but I'm not sure what kind of methods are appropriate. Thanks for raising out subsequence mining! :) $\endgroup$ – InvadersMustDie Sep 4 at 2:21

Yes, K-means is an option. But you need to do some data processing first.

  • My first concern is about quantifying those statuses (are they linearly depended, like does B means something in the middle of A and C)? then numerate them (A to 1 and B to 2 and C to 3 etc)
  • Represent your data set in a dimensional way, melt years columns into three columns

    1. Year (with values of 2000, 2001, 2003 etc) so you get a time series dimension.
    2. Year_status (status value).
    3. Year_percentage (percentage of the year).
  • Try to treat pattern by moving window (kernel), and try different sizes.. for example size of 3 will cluster every 3 followed statuses into patterns centroids (for example.. having a common occurrence of A, E, A)

  • $\begingroup$ Thanks for the suggestion! I was worried about the statuses, because they aren't linearly related. Which is why I thought k-means may not be a good method. Also, thanks for raising the idea of a moving window, I will explore that possibility! $\endgroup$ – InvadersMustDie Sep 4 at 2:22
  • $\begingroup$ Then, think about it as a semantic data.. where statuses are independent words.. back to the K-means, medium.com/@MSalnikov/… is an example of using kmeans with semantic analysis (by tf-idf). One thing to note, with tfidf, the moving window can be used by the ngram_range variable of the tfidf. $\endgroup$ – krayyem Sep 4 at 3:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.