1
$\begingroup$

I have trained a network to find the similarity between two images. The test dataset contain equal number of similar and dissimilar samples. Each class has approx. 13822 samples.

I tried different threshold, for example, in the last layer of the network I used sigmoid activation function therefore the output is between [0,1].

I tested threshold such as from 0.5 to 0.9 (for example, if the prediction score is greater then 0.5/0.6/,....,/0.9, consider it as positive sample else negative sample).

Based on this strategy I got the following results:

Threshold   TP  TN  FP  FN  Accuracy    Precision   Recall  F1
0.5 --> 12570   13804   18  1252    0.95    1.00    0.91    0.95
0.6 --> 12115   13813   9   1707    0.94    1.00    0.88    0.93
0.7 --> 11451   13819   3   2371    0.91    1.00    0.83    0.91
0.8 --> 10124   13822   0   3698    0.87    1.00    0.73    0.85
0.9 --> 5132    13822   0   8690    0.69    1.00    0.37    0.54

What I am interested in is to find the optimal threshold based on some method which separates positive and negative classes.

$\endgroup$

1 Answer 1

1
$\begingroup$

Rank all your instances by their output value. Then for each instance (or for each different output value, if many instances have the same value), calculate the performance (precision, recall, f-score) considering that the threshold is the output value for this instance. When this is done for all the instances, you can plot the graph of f-score as a function of the threshold for a nice visual result, or simply pick the threshold corresponding to the maximum performance.

For the record, this is also how a ROC curve can be built manually.

Btw your optimal threshold is certainly lower than 0.5, since your precision is 1.0 at 0.5 but you can probably still increase recall.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.