0
$\begingroup$

I am working on a binary classification problem using random forests (75:25 - class proportion). Label 0 is minority class. So, I am following the below approach

a) execute RF with default hyperparameters

b) execute RF with best hyperparameters (GridsearchCV with stratified K fold and scoring was F1)

While with default hyperparameters, my train data was overfit as can be seen from the results below. But, I went ahead and tried the default paramters in test data as well (results below)

enter image description here

default hyperparameters - Test data confusion matrix and classification report

enter image description here

enter image description here

Best hyperparameters - Test confusion matrix and classification report

enter image description here

enter image description here

So, my question is

a) Why does a best parameters lead to drop in test data performance? Despite my model.best_score_ returning 86.5 as f1-score? I thought f1 scoring would allow us to find the best f1-score for both the classes. Looks like it is only focusing on class 1. How can I make the score function to work to increase the f1-score for minority class?

b) This makes me feel like it is okay to stick with the overfit model as it provides me relatively good performance on test data (when compared to best parameter model because it performs poorly)

c) My objective is to maximize the metrics like recall and precision for label 0 (minority class)? How can I do that? Any suggestions please?

d) In this case, should I go ahead with the overfit model with default parameters?

update

when I invert the labels based on below answer, meaning 0's as 1's and 1's as 0's, I get the below performance

enter image description here

update - code for best hyparameters

from sklearn.model_selection import GridSearchCV
param_grid = { 
    'n_estimators': [100,200,300,500],
    'max_features': ['auto', 'sqrt', 'log2'],
    'max_depth' : [4,5,6,7,8],
    'criterion' :['gini', 'entropy']
}
skf = StratifiedKFold(n_splits=10, shuffle=False)
model = GridSearchCV(rfc,param_grid=None,cv = skf, scoring='f1')
model.fit(ord_train_t, y_train)
print(model.best_params_)
print(model.best_score_)
rfc = RandomForestClassifier(random_state=42, max_features='sqrt', n_estimators= 500, max_depth=8, criterion='gini')
rfc.fit(ord_train_t, y_train)
y_train_pred = rfc.predict(ord_train_t)
y_test_pred = rfc.predict(ord_test_t)
y_train_proba = rfc.predict_proba(ord_train_t)
y_test_proba = rfc.predict_proba(ord_test_t)

code for default hyparameters

rfc = RandomForestClassifier()
rfc.fit(ord_train_t, y_train)
y_train_pred = rfc.predict(ord_train_t)
y_test_pred = rfc.predict(ord_test_t)
y_train_proba = rfc.predict_proba(ord_train_t)
y_test_proba = rfc.predict_proba(ord_test_t)
$\endgroup$

1 Answer 1

1
$\begingroup$

A simple hack, minority class people usually keep this as 1. As most metrics take into account that only. So you can chnage the mapping that is 0 to 1 and 1 to 0 and see if it works.

I think its happening as model is optimising for 1s which in this case is not aligned with what you want to do

$\endgroup$
12
  • $\begingroup$ Oh wow..Understand. Let me try and update you. Thanks for your help. Will update you here. upvoted $\endgroup$
    – The Great
    Feb 8, 2022 at 9:39
  • $\begingroup$ Isn't there any other method to optimize the metrics for minority class? I thought it should just be simple. Didn't know we had to change the label itsef $\endgroup$
    – The Great
    Feb 8, 2022 at 9:42
  • $\begingroup$ Not Sure, how its implemented... I have never seen anything like that..But that being said needs more exploration.. $\endgroup$ Feb 8, 2022 at 9:57
  • $\begingroup$ Updated the post with new output after inverting labels $\endgroup$
    – The Great
    Feb 8, 2022 at 10:15
  • $\begingroup$ You recall has improved a little but still model needs a lot of imrpovement. I would suggest to do upsampling/downsampling while training or use class weights $\endgroup$ Feb 8, 2022 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.