how can I calculate the y_score for a roc_auc_score?

I have a classifier, for classes {0,1}, say RandomForestClassifier.

Then, when I apply it to my test data, I will get a list of {0,1}

But roc_auc_score expects y_true and y_score.

As dummy as it might look, after fitting the model, I was making the following:

def find_prob(df):
    def prob(token):
        # the mean is the probability of label {1}
        return df[df.token == token].predicted.mean()
    df['predicted_score'] = df.token.map(prob)
    return df

X_val['predicted'] = model.predict(X_val).astype(int)
X_val = find_prob(tokenize(X_val))

roc_auc_score(y_val, X_val.predicted_score)

where tokenize is only a function which concatenate all the values in a row to a "unique" token string, so that I would be able of finding all instances of a token and check how many were labeled as {0} and how many were labeled as {1}.

Is this the correct approach? Thank you very much.


y_score can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions

y_score = model.predict_proba(x)[:,1]

AUC = roc_auc_score(y, y_score) # Above 0.5 is good
| improve this answer | |

So, I have found my answer, Now I am going to share it. y_score is the prediction of the x_test, where you can say in my dataset, it is images, in dataset_train. You can get the prediction of the dataset using this -

p = torch.nn.functional.softmax(output, dim=1)
prediction = torch.argmax(p, dim=1)

Where p is the probability and then using argmax, you can get what you want.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.