# How to tune / choose the preference parameter of AffinityPropagation?

I have large dictionary of "pairwise similarity matrixes" that would look like the following:

similarity['group1']:

array([[1.        , 0.        , 0.        , 0.        , 0.        ],
[0.        , 1.        , 0.09      , 0.09      , 0.        ],
[0.        , 0.09      , 1.        , 0.94535157, 0.        ],
[0.        , 0.09      , 0.94535157, 1.        , 0.        ],
[0.        , 0.        , 0.        , 0.        , 1.        ]])


In short, every element of the previous matrix is the probability that record_i and record_j are similar (values being 0 and 1 inclusive), 1 being exactly similar and 0 being completely different.

I then feed each similarity matrix into an AffinityPropagation algorithm in order to group / cluster similar records:

sim = similarities['group1']

clusterer = AffinityPropagation(affinity='precomputed',
damping=0.5,
max_iter=25000,
convergence_iter=2500,
preference=????)) # ISSUE here

affinity = clusterer.fit(sim)

cluster_centers_indices = affinity.cluster_centers_indices_
labels = affinity.labels_


However, since I run the above on multiple similarity matrixes, I need to have a generalised preference parameter which I can't seem to tune.

It says in the docs that it's by default set as the median of the similarity matrix, however I get lots of false positives with this setup, the mean sometimes work sometimes gives too many clusters etc...

e.g: when playing with the preference parameter, these are the results I get from the similarity matrix

• preference = default # which is the median (value 0.2) of the similarity matrix: (incorrect results, we see that the record 18 shouldn't be there because the similarity with the other records is very low):

 # Indexes of the elements in Cluster n°5: [15, 18, 22, 27]

{'15_18': 0.08,
'15_22': 0.964546229533378,
'15_27': 0.6909703138051403,
'18_22': 0.12,    # Not Ok, the similarity is too low
'18_27': 0.19,    # Not Ok, the similarity is too low
'22_27': 0.6909703138051403}

• preference = 0.2 in fact from 0.11 to 0.26: (correct results as the records are similar):

 # Indexes of the elements in Cluster n°5: [15, 22, 27]

{'15_22': 0.964546229533378,
'15_27': 0.6909703138051403,
'22_27': 0.6909703138051403}


My question is: How should I choose this preference parameter in a way that would generalise?