Creating better features for clustering

Question

I am trying clustering for the first time trying to separate my user into three categories (or the categories I though that they will should fall in).

First of all I have two tables that describe states and times. The state table looks like that:

    [[7059992, 7082352, 7083172, 7085480, 7095742],
     [7443343, 7453591, 7463815, 7474055, 7484295],
     [3122958, 3133217, 3143449, 3153658, 3163918], 

each row is a user going through some states. This type of features can not be used for clustering directly so I had to create some features.

What I did was to mark the no change between states with a zero and any change with a timestamp between the last step when a transition was made.

so this gave me a table like that

    [ 3205.,     0.,     0.,     0.],
    [    0.,     0.,     0.,     0.],
    [10005., 16620.,  4626., 26503.],

so user 1 goes through 3205,0,0,0 which means that at the beginning there was a transition that happened 3205ms time before the last change.

user 2 goes through 0,0,0,0 that means that no change happened

and user 3 goes through 10005, 16620, 4626, 26503 which means that the user went through four changes with their respective timestamps.

I thought then that these can be now features for clustering and I tried k-means to get some labels. (I also did normalization to reduce the effects of zero values). Last column is the assigned cluster id.

 0.567415  0.347368  0.673178  0.185788         1  
 0.000000  0.000000  0.000000  0.000000         0  
 0.413984  0.000000  0.000000  0.439690         2  

I want my clustering technique to try to find three categories that have these distinct characteristics.

• First class: no transitions, pure zeros

• Second class: few transitions (one or max two nonzero values, but each of them is a large value.)

• Third class: many transitions (three or four non zero values, but small values for each of them).

K-means seems to merge many time second and third class since the zeros and the large values seem many times the same with many transitions and small values.

If there any clustering technique that can work better with this type of data? If not can I also denote the no transition instead of 0, with some other value, like NaN? That might help to differentiate better the second and third class.

Answer 1

Setting missing values to 0 does not work well. Because this causes huge deviations.

K-means cannot handle NaN values well, at least in the standard implementations. You could remove the all zero vectors, and then use a nan-toletant implementation, but this will likely also cause undesired effects. Because the absence of a transaction is not the same as an unknown timestamp.

In the end, I believe you have not understood your problem yet. K-means likely is the wrong tool, so it does not matter how you prepare the data - it's answering a different question, but you haven't understood what you are asking yet (and no, "what are the clusters" is not a well defined question - phrase it mathematically, then you have at least a formal version of the question).