1
$\begingroup$

I have made an ANN in Keras that works on an imbalanced binary dataset. The data is used after fitting the model to predict the binary classes and I want to choose a threshold s.t. sensitivity and specificity is maximized.

This is the code that I am using right now, iterating through all thresholds from 0-1 and finding the best one using G-mean score.

predictions = model_p.predict(Xt)
thresholds = arange(0, 1, 0.001)
threshold = -1
best_Gscore = 0
false_positive = 0
true_positive = 0
false_negative = 0
true_negative = 0

    for z in thresholds:
        print("Threshold => %f " % (z))
        fp = 0
        fn = 0
        tp = 0
        tn = 0
        for i in range(len(yt)):
            if( yt[i] == 0 and predictions[i] > z ):
                fp += 1
            elif( yt[i] == 1 and predictions[i] > z ):
                tp += 1
            elif( yt[i] == 1 and predictions[i] <= z ):
                fn += 1
            elif( yt[i] == 0 and predictions[i] <= z ):
                tn += 1
        
        if( (tp+fn) == 0):
            continue
        if( (tn+fp) == 0):
            continue
        TPR = fp / (fp + tn)
        #sens = tp / (tp + fn)
        #spec = tn / (tn + fp)
        FPR = tp / (tp + fn)
        Gscore = math.sqrt(TPR*(1-FPR))

        print("J Stat => %f " % (Gscore), flush=True)

        if( Gscore > best_Gscore ):
            best_Gscore = Gscore
            false_positive = fp
            false_negative = fn
            true_positive = tp
            true_negative = tn
            threshold = z

But is there a better way to maximize sens and spec? Perhaps finding a sens and spec suchs that

| sens - spec | < 0.05 and sens*spec > score_max

Then once this score_max is found you can run through smaller jumps for like +- 0.2 on both? Or is there another way to find sensitivity and specificity maximum?

$\endgroup$

1 Answer 1

1
$\begingroup$

It's impossible in general to optimize both sensitivity and specificity, in the sense of finding a threshold for which sensitivity is maximum and specificity is maximum:

  • sensitivity is high when TP is high and FN is low
  • specificity is high when TN is high and FP is low

But since:

  • When the threshold is increased, more instances are predicted negative so TN and FN increase, TP and FP decrease.
  • When the threshold is decreased, more instances are predicted positive so TP and FP increase, TN and FN decrease.

Therefore one cannot have the lowest possible FP and the lowest possible FN at the same time.

In other words, max sensitivity is when all instances are predicted positive while max specificity is when all instances are predicted negative. Clearly both are not compatible.

Instead one can only optimize a combination of both, similarly to F-score which is the harmonic mean of precision (related to specificity) and recall (sensitivity).

$\endgroup$
4
  • $\begingroup$ Thank you for your explanation! So there is no approximate algorithm for finding a "reasonable" local maximum for sensitivity and specificity other than G-mean or F-score for example ? $\endgroup$
    – Don_twice
    Commented May 5, 2022 at 14:06
  • $\begingroup$ It depends what you call "reasonable" :) actually it depends what is defined as a local maximum for two values. It must be some form of mean, because the two values in this case cannot both be maximum with the same threshold. $\endgroup$
    – Erwan
    Commented May 5, 2022 at 18:24
  • $\begingroup$ I understand what you mean. I'm also guessing given a maximum G-mean score on all thresholds there is not another score with "better" marks on sensitivity and specificity but perhaps only different values? I'm thinking something like finding the G-mean then searching in that neighborhood for a value where sens and spec are closer. You could also save all G-mean scores in a list that are say 0.1 from the higest score, search all those scores within their respective neighborhoods if there is a sensitivity and specificity that is higher, but perhaps this is not the way. $\endgroup$
    – Don_twice
    Commented May 6, 2022 at 7:45
  • $\begingroup$ It's mathematically impossible to increase both by changing the threshold: in the best case one of the two would increase and the other would stay the same. And since there is only a finite number of possible thresholds for a dataset, by picking the maximum mean (for example), it's impossible to find another threshold which would do better for any the two. $\endgroup$
    – Erwan
    Commented May 7, 2022 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.