How to tune / choose the preference parameter of AffinityPropagation?

Question

I have large dictionary of "pairwise similarity matrixes" that would look like the following:

similarity['group1']:

array([[1.        , 0.        , 0.        , 0.        , 0.        ],
       [0.        , 1.        , 0.09      , 0.09      , 0.        ],
       [0.        , 0.09      , 1.        , 0.94535157, 0.        ],
       [0.        , 0.09      , 0.94535157, 1.        , 0.        ],
       [0.        , 0.        , 0.        , 0.        , 1.        ]])

In short, every element of the previous matrix is the probability that record_i and record_j are similar (values being 0 and 1 inclusive), 1 being exactly similar and 0 being completely different.

I then feed each similarity matrix into an AffinityPropagation algorithm in order to group / cluster similar records:

sim = similarities['group1']

clusterer = AffinityPropagation(affinity='precomputed', 
                                damping=0.5, 
                                max_iter=25000, 
                                convergence_iter=2500, 
                                preference=????)) # ISSUE here

affinity = clusterer.fit(sim)

cluster_centers_indices = affinity.cluster_centers_indices_
labels = affinity.labels_

However, since I run the above on multiple similarity matrixes, I need to have a generalised preference parameter which I can't seem to tune.

It says in the docs that it's by default set as the median of the similarity matrix, however I get lots of false positives with this setup, the mean sometimes work sometimes gives too many clusters etc...

---

e.g: when playing with the preference parameter, these are the results I get from the similarity matrix

• preference = default # which is the median (value 0.2) of the similarity matrix: (incorrect results, we see that the record 18 shouldn't be there because the similarity with the other records is very low):

   # Indexes of the elements in Cluster n°5: [15, 18, 22, 27]
  
   {'15_18': 0.08,
   '15_22': 0.964546229533378,
   '15_27': 0.6909703138051403,
   '18_22': 0.12,    # Not Ok, the similarity is too low
   '18_27': 0.19,    # Not Ok, the similarity is too low
   '22_27': 0.6909703138051403}
  

• preference = 0.2 in fact from 0.11 to 0.26: (correct results as the records are similar):

   # Indexes of the elements in Cluster n°5: [15, 22, 27]
  
   {'15_22': 0.964546229533378,
   '15_27': 0.6909703138051403,
   '22_27': 0.6909703138051403}
  

---

My question is: How should I choose this preference parameter in a way that would generalise?

Answer 1

One option is to systematically try different values of the preference (hyper)parameter and since which value generalized best by measuring evaluation metric performance on a hold-out dataset. You mention false positives so that can be used as the evaluation metric. This process is typically called cross-validation.