2
$\begingroup$

I was wondering if it makes sense to apply clustering techniques on an aggregation of data, like, I have three different sources of data such as S1 S2 and S3 where each of these sources share some common columns but the majority are not shared. Does it make sense to group all the sources with all the columns in a big dataframe and apply clustering techniques whereas some records will only have null values for the columns that are not part of its corresponding service. Thank you.

Update: The input is basically logs coming from different services with different columns where some of them are shared between all services. The output should be a cluster of records representing a user having the same behavior.
The logs are gathered by second corresponding of a user action. They are aggregated by hours for each user to derive other features (the granularity of seconds it according to me, too much).
The goal, is to detect anormal behaviors.
And the question is, can I regroup different dataframe having different features while some of them are shared and run a K-mean on it. Because my dataframe would look like this (s* for service, and common_ for common feature):

-------------------------------------------------------------
s1_f1 | s1_f2 | s2_f1 | s2_f2 | s3_f1 | common_f1 | common_f2
  1       OK     NULL    NULL    NULL     midday      less
  2       OK     NULL    NULL    NULL     midnight    more
 NULL    NULL     2       5      NULL     midday      less
 NULL    NULL     8       9      NULL     morning     less
 NULL    NULL    NULL    NULL    777      morning     more
 NULL    NULL    NULL    NULL    888      night       more
$\endgroup$
3
  • $\begingroup$ What would be the output, and how sould you put it into use? There is no use in just doing something without a plan. $\endgroup$ Apr 13 '16 at 21:09
  • $\begingroup$ @marcL, You mentioned data "aggregated", which implies some type of summarization -- or roll-up -- of the data. However, based on the details you've provided, it appears that you're just stacking three data sets that came from different sources. Can you clarify? $\endgroup$
    – Vishal
    Apr 13 '16 at 21:31
  • $\begingroup$ @Vishal I updated my question, hope its more clear now $\endgroup$ Apr 14 '16 at 21:46
0
$\begingroup$

Since some features are missing for specific sources, the missing values are not missing-at-random but are systematically missing. In this situation, I'd advise against doing clustering on the combined data set with all available features. If missing values were occurring at random, you could have used some missing value imputation method before performing cluster analysis. However, since the values are systematically missing, imputation would be difficult to tackle. (You could try to predict those missing values, but I am afraid that will add a lot of unnecessary noise in the data.)

I'd recommend choosing from one of these two options:

  1. Perform clustering on the combined data set, but use only those features that are non-missing across all sources.
  2. Perform three different cluster analysis, one for each source. This way, you can ensure that you are using as many features (information) as possible. The determination of "abnormal" behavior can then be determined within each source. This can be an added benefit since it would allow you to be more specific about why a use might be abnormal, as you have more features that can be used to explain this. The results can also be then summarized across all sources to create one consolidated report.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.