How to calculate y_score for ROC AUC?

Question

how can I calculate the y_score for a roc_auc_score?

I have a classifier, for classes {0,1}, say RandomForestClassifier.

Then, when I apply it to my test data, I will get a list of {0,1}

But roc_auc_score expects y_true and y_score.

As dummy as it might look, after fitting the model, I was making the following:

def find_prob(df):
    def prob(token):
        # the mean is the probability of label {1}
        return df[df.token == token].predicted.mean()
    df['predicted_score'] = df.token.map(prob)
    return df

X_val['predicted'] = model.predict(X_val).astype(int)
X_val = find_prob(tokenize(X_val))

roc_auc_score(y_val, X_val.predicted_score)

where tokenize is only a function which concatenate all the values in a row to a "unique" token string, so that I would be able of finding all instances of a token and check how many were labeled as {0} and how many were labeled as {1}.

Is this the correct approach? Thank you very much.

Answer 1

y_score can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions

y_score = model.predict_proba(x)[:,1]

AUC = roc_auc_score(y, y_score) # Above 0.5 is good

Answer 2

So, I have found my answer, Now I am going to share it. y_score is the prediction of the x_test, where you can say in my dataset, it is images, in dataset_train. You can get the prediction of the dataset using this -

p = torch.nn.functional.softmax(output, dim=1)
prediction = torch.argmax(p, dim=1)

Where p is the probability and then using argmax, you can get what you want.