1
$\begingroup$

I have large dictionary of "pairwise similarity matrixes" that would look like the following:

similarity['group1']:

array([[1.        , 0.        , 0.        , 0.        , 0.        ],
       [0.        , 1.        , 0.09      , 0.09      , 0.        ],
       [0.        , 0.09      , 1.        , 0.94535157, 0.        ],
       [0.        , 0.09      , 0.94535157, 1.        , 0.        ],
       [0.        , 0.        , 0.        , 0.        , 1.        ]])

In short, every element of the previous matrix is the probability that record_i and record_j are similar (values being 0 and 1 inclusive), 1 being exactly similar and 0 being completely different.

I then feed each similarity matrix into an AffinityPropagation algorithm in order to group / cluster similar records:

sim = similarities['group1']

clusterer = AffinityPropagation(affinity='precomputed', 
                                damping=0.5, 
                                max_iter=25000, 
                                convergence_iter=2500, 
                                preference=????)) # ISSUE here

affinity = clusterer.fit(sim)

cluster_centers_indices = affinity.cluster_centers_indices_
labels = affinity.labels_

However, since I run the above on multiple similarity matrixes, I need to have a generalised preference parameter which I can't seem to tune.

It says in the docs that it's by default set as the median of the similarity matrix, however I get lots of false positives with this setup, the mean sometimes work sometimes gives too many clusters etc...


e.g: when playing with the preference parameter, these are the results I get from the similarity matrix

  • preference = default # which is the median (value 0.2) of the similarity matrix: (incorrect results, we see that the record 18 shouldn't be there because the similarity with the other records is very low):

     # Indexes of the elements in Cluster n°5: [15, 18, 22, 27]
    
     {'15_18': 0.08,
     '15_22': 0.964546229533378,
     '15_27': 0.6909703138051403,
     '18_22': 0.12,    # Not Ok, the similarity is too low
     '18_27': 0.19,    # Not Ok, the similarity is too low
     '22_27': 0.6909703138051403}
    
  • preference = 0.2 in fact from 0.11 to 0.26: (correct results as the records are similar):

     # Indexes of the elements in Cluster n°5: [15, 22, 27]
    
     {'15_22': 0.964546229533378,
     '15_27': 0.6909703138051403,
     '22_27': 0.6909703138051403}
    

My question is: How should I choose this preference parameter in a way that would generalise?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.