1
$\begingroup$

I have created (in python) the code for a Random Forest classification model for a labeled dataset using sklearn. The model works very well.

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, confusion_matrix, precision_score, recall_score, ConfusionMatrixDisplay
from sklearn.model_selection import RandomizedSearchCV, train_test_split
from scipy.stats import randint

# Data Processing
import pandas as pd
import numpy as np

## load dataset
    col_names = ['id', 'inputdatetime', 'ax1', 'ay1', 'az1', 'gx1', 'gy1', 'gz1', 'ax2', 'ay2', 'az2', 'gx2', 'gy2', 'gz2', 'ax3', 'ay3', 'az3', 'gx3', 'gy3', 'gz3', 'ax4', 'ay4', 'az4', 'gx4', 'gy4', 'gz4', 'ax5', 'ay5', 'az5', 'gx5', 'gy5', 'gz5', 'label']
    pima = pd.read_csv(r"C:\Users\danie\Desktop\Testes Tese\rh_all_final.csv", header=None, names=col_names, delimiter=';', skiprows = 1)
    pima = pima.drop('id', axis=1)
    pima = pima.drop('inputdatetime', axis=1)
    pima['label'] = pima['label'].map({'A':1,'B':2,'C':3,'D':4,'E':5,'F':6,'G':7,'H':8,'I':9,'J':10,'K':11,'L':12,'M':13,'N':14,'O':15,'P':16,'Q':17,'R':18,'S':19,'T':20,'U':21,'V':22,'W':23,'Y':24,'X':25,'Z':26,'DAR':27,'PEGAR':28,'ANDAR':29,'PARAR':30,'PULAR':31,'ESCREVER':32,'COMPRAR':33,'PERGUNTAR':34,'IGNORAR':35,'CASA':36,'TELEMOVEL':37,'CARRO':38,'BARCO':39,'COMPUTADOR':40,'LIVRO':41,'CHAVE':42,'CAMA':43,'CADEIRA':44,'IR':45,'TV':46})

    ## Split the data into features (X) and target (y)
    X = pima.drop('label', axis=1)
    y = pima['label']

    ## Split the data into training and test sets
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
    rf = RandomForestClassifier()
    rf.fit(X_train, y_train)
    y_pred = rf.predict(X_test)

Now I need to sort data that is generated after creating the model and has no label. I can do that too. My question is regarding how to get the confidence in the classification made.

Is there any sklearn methods that calculat a metric that I can use to know the degree of confidence in the classification of new unlabeled data?

Thanks for the help!

I tried looking at the accuracy_score, precision_score and recall_score methods, but I think they only work for labeled data. Or am I mistaken?

$\endgroup$
1
  • $\begingroup$ Try using the predict_proba method of the random forest classifier. $\endgroup$
    – Oxbowerce
    Commented Aug 15, 2023 at 16:24

1 Answer 1

1
$\begingroup$

This is a very interesting question, which requires first to distinguish some concepts:

1. Uncertainty of the outcome / Calibration

This is what you measure with predict_proba and it tells you with which probability the model expects which outcome to occur. Imagine that you build a model to predict whether it will rain on the afternoon, based on three features (temperature, is_cloudy, is_windy) measured in the morning (just as an example, we do not need much realism, here). Probably a cloudy sky in the morning will cause the model to give a higher probability of rain in the afternoon. This prediction is done well, if a predicted probability of 75% will in 75% of the cases occur with rain. We call a model calibrated, when this predicted probability approximates the observed probability well

Unfortunately, we need to know the actual outcome to measure the calibration of a model.

2. Quality of the prediction

This tells us how good or useful our predictions are. Let's take the example from above. Imagine 3 models: one just know it rains (in the area of interest) at 30% of the days and gives a constant output of 0.3, independent of the features. The second one just uses is_cloudy as feature and the third one is a more complex model using all features. All three might be calibrated, but the third one might give much better predictions.

For such a model, we would use accuracy, precision, recall, f1-score or similar values for binary preditions (via predict) or the area under the roc-curve / precision-recall-curve for probabilistic predictions (via predict_proba).

Unfortunately, this again requires labeled data to know how good the predictions are

Confidence

Confidence is a statistical property that one might know from confidence intervals. This can measure how much the predictions vary, if other training data is used. If you train your model on 500 samples, then the model and predictions might be different, depending on which 500 samples you take.

Luckily, this confidence of the prediction can be measured without labeled data. As a start, you could use k-fold cross validation and compare the predictions of the $k$ models on you hold-out dataset. This would give you a variance for each sample and a mean variance of predictions.

Although technically not exactly a confidence interval, it would give you a good indication how much the predictions depend on the selection of training data. A high variance means a low confidence in the prediction as it base on a lot of sampling noise.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.