1
$\begingroup$

I have logs of user activity on my system. This is a CMS system.

The logs consist of:

  • User ID
  • Action Performed (one of 110 possible actions - things like page edited, page read, login, etc.)
  • Site action was performed on (not relevant for all possible actions)
  • Object action was performed on (document name or document id - not relevant for all possible actions)
  • Date and time that the action was performed

I would like to use this data to cluster users into groups with similar behavior, so that I can then survey the groups to find out who they are and then target them with training interventions, newsletters etc.

I need help with converting the above list of things into clusters.

What techniques can I use to generate a matrix from the above data that I can feed into a K-Means cluster algorithm?

I tried counting action by user and using that to feed a cluster algorithm but the result was useless. I also tried treating the site and document ids as text, feature hashing that and feeding it into a K-Means cluster model, also got useless data out.

Improve this question
$\endgroup$
8
  • $\begingroup$ "result was useless" how do you know? K-means is unsupervised so there is no intrinsic measurement of fitness. $\endgroup$
    – Louis T
    Nov 8, 2017 at 23:52
  • $\begingroup$ Learn an embedding for each user based on their activity. Since you have time series data, treat it as a sequence prediction problem with the user embedding as a feature or initial state. I know this description is terse so I hope someone can fill in the blanks if you need it. Welcome to the site and good luck! $\endgroup$
    – Emre
    Nov 9, 2017 at 0:23
  • $\begingroup$ @LouisT for my action count matrix I ended up with a huge circular cluster containing a tiny dot of another cluster in the middle. I think there is something technically wrong with how I fed the data to the K-Means cluster algorithm, but I don't know what that might be. $\endgroup$
    – Joon
    Nov 9, 2017 at 8:26
  • $\begingroup$ @Emre thanks that is an interesting approach. A bit beyond my capabilities though so maybe someone will be friendly enough to help get me started ;-) $\endgroup$
    – Joon
    Nov 9, 2017 at 8:27
  • $\begingroup$ @Joon I don't understand what do you mean by "circular cluster". I am imaging a data frame, with each row represents a user and the 110 columns each represents an action. and the values of the data frame is the count of actions performed by the users. $\endgroup$
    – Louis T
    Nov 9, 2017 at 11:05

1 Answer 1

Reset to default
1
$\begingroup$

Scaling

Standardize you data by $(x - mean(x)) / std(x)$

Most K-mean implementation by default uses Euclidean distance which assumes the equal importance of all features. This requires proper scaling to prevent one action dominates the others.

Dimensionality

K-mean is not robust against the curse of dimensionality (see this post). So as always it good to reduce you dimensionality before feed it to the algorithm.

Do some feature engineering first. For example, login and page load can be group together as a measure of passive engagement, page edit and page created could be considered as a single activate engagement feature.

Also, you can try using some standard dimensionality reduction algorithm like PCA

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks Louis. One last question about the standardization for optimal clustering: Should I standardize all cells of my matrix, or only within a feature? $\endgroup$
    – Joon
    Nov 10, 2017 at 8:52
  • 1
    $\begingroup$ @Joon standardize within each feature, such that each feature has zero mean and one as stdev $\endgroup$
    – Louis T
    Nov 10, 2017 at 8:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.