1
$\begingroup$

I have data which contains access duration of some items.

Example:
t0~t5 is the access time duration, 1 means the items was accessed in the time duration, 0 means it wasn't.

ID,t0,t1,t2,t3,t4
0,0,0,1,1,1
1,0,1,1,1,1
2,0,1,1,0,0
3,1,1,0,0,1
4,1,1,0,0,1

In the above example, groups ID=0,1 are what I want.

ID=3,4 aren't because their distance is short but they are not continuous.

I tried KMeans and DBSCAN, they all cluster ID=3,4 as one group and it makes sense. But it doesn't do what I want.

Is there any possible pre-processing of data to reach what I want ?

Or I should use other analytic tool?

$\endgroup$
1
$\begingroup$

What might help is a custom distance computation as input to the clustering algorithm. These algorithms usually take Euclidean distance as a measure of dissimilarity.

You can try DBSCAN (in Python scikit-learn), with metric='precomputed' and 'X' as a custom distance matrix. You can construct this distance matrix to conform to your requirement. Eg: specify that nodes 3 and 4 have a large distance, even though they are equal.

| improve this answer | |
$\endgroup$
  • $\begingroup$ What does "custom distance matrix" mean? thx $\endgroup$ – code_worker Mar 30 '19 at 14:45
  • $\begingroup$ This is the matrix of pairwise distances that you will compute in the way that you want. Eg: 3 and 4 will have a large distance. You then pass this as input to the clustering algorithm. $\endgroup$ – raghu Mar 31 '19 at 4:16
  • $\begingroup$ So I need to define the matrix myself. thx. I originally thought there's a common way to define it. $\endgroup$ – code_worker Mar 31 '19 at 8:26
0
$\begingroup$

It is probably worth to transform the data, such that each record is "continuous" (call it differently - for example "contiguous" because the term continuous has a widely known mathematical meaning), and if necessary make multiple copies.

K-means minimizes the sum of squares. I don't see how that is beneficial here.

There is generalized DBSCAN. You can define arbitrary neighbor predicates for it. For example, you could the define that neighbors (candidates for merging into the same cluster) must have a contiguous overlap of at least two active timepoints. Then consider whether this satisfies your notion of clusters because of the transitivity computed by DBSCAN.

My guess is that you'll rather want to e.g. extract all contiguous subsequences of a minimum length - say, 2 - of all records and simply count them to identify the most frequent subsequences. If you implement this with an efficient bit representation, then it will be very fast.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Now I don't know how to transform data and cluster it to reach what I want. $\endgroup$ – code_worker Apr 1 '19 at 1:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.