I am working on a binary classification problem using random forests (75:25 - class proportion). Label 0 is minority class. So, I am following the below approach
a) execute RF with default hyperparameters
b) execute RF with best hyperparameters (GridsearchCV with stratified K fold and scoring was F1
)
While with default hyperparameters, my train data was overfit as can be seen from the results below. But, I went ahead and tried the default paramters in test data as well (results below)
default hyperparameters - Test data confusion matrix and classification report
Best hyperparameters - Test confusion matrix and classification report
So, my question is
a) Why does a best parameters lead to drop in test data performance? Despite my model.best_score_ returning 86.5 as f1-score
? I thought f1
scoring would allow us to find the best f1-score
for both the classes. Looks like it is only focusing on class 1
. How can I make the score function to work to increase the f1-score
for minority class?
b) This makes me feel like it is okay to stick with the overfit model as it provides me relatively good performance on test data (when compared to best parameter model because it performs poorly)
c) My objective is to maximize the metrics like recall and precision for label 0 (minority class)? How can I do that? Any suggestions please?
d) In this case, should I go ahead with the overfit model with default parameters?
update
when I invert the labels based on below answer, meaning 0's as 1's and 1's as 0's, I get the below performance
update - code for best hyparameters
from sklearn.model_selection import GridSearchCV
param_grid = {
'n_estimators': [100,200,300,500],
'max_features': ['auto', 'sqrt', 'log2'],
'max_depth' : [4,5,6,7,8],
'criterion' :['gini', 'entropy']
}
skf = StratifiedKFold(n_splits=10, shuffle=False)
model = GridSearchCV(rfc,param_grid=None,cv = skf, scoring='f1')
model.fit(ord_train_t, y_train)
print(model.best_params_)
print(model.best_score_)
rfc = RandomForestClassifier(random_state=42, max_features='sqrt', n_estimators= 500, max_depth=8, criterion='gini')
rfc.fit(ord_train_t, y_train)
y_train_pred = rfc.predict(ord_train_t)
y_test_pred = rfc.predict(ord_test_t)
y_train_proba = rfc.predict_proba(ord_train_t)
y_test_proba = rfc.predict_proba(ord_test_t)
code for default hyparameters
rfc = RandomForestClassifier()
rfc.fit(ord_train_t, y_train)
y_train_pred = rfc.predict(ord_train_t)
y_test_pred = rfc.predict(ord_test_t)
y_train_proba = rfc.predict_proba(ord_train_t)
y_test_proba = rfc.predict_proba(ord_test_t)