How to set the priority to Machine leaning algorithms for Binary classification among Four based on accuracy and fitting

Question

Rain Classification in Australia

Under this context, sklearn classification algorithms will be used, namely:

Logistic Regression Classification (Parametric) Decision Tree Classification (Non parametric) Random Forest Classification (Non parametric) K-Nearest Neighbour (KNN) Classification (Non parametric)

Train Test Split is 80-20

Accuracy Score :

Logistic Regression Test Score     0.854062
Logistic Regression Train Score    0.853797

Decision Tree Test Score           0.795838
Decision Tree Train Score          1.000000

Random Forest Test Score           0.858269
Random Forest Train Score          0.999978

K-Nearest Neighbour Test Score     0.817180
K-Nearest Neighbour Train Score    0.831138

`Null accuracy score: 0.7815`

Logistic Regression is performing good without overfitting. But if look at accuracy ,Random Forest accuracy is better.

How to check for underfitting.

Confusion Matrix

ROC AUC :

ROC AUC For LR : 0.8742

ROC AUC For DT : 0.7072

ROC AUC For RF : 0.8883

ROC AUC For KNN : 0.7928

Classification Metrices

Classification accuracy : 0.8583
Classification error : 0.1417
Precision : 0.9586
Recall : 0.8726
True Positive Rate : 0.8726
False Positive Rate : 0.2286
Specificity : 0.7714

Logistic Regression

             precision    recall  f1-score   support

          No       0.88      0.95      0.91     17650
         Yes       0.74      0.52      0.61      4935

    accuracy                           0.85     22585
   macro avg       0.81      0.73      0.76     22585
weighted avg       0.85      0.85      0.84     22585

Decision Tree

           precision    recall  f1-score   support

          No       0.87      0.86      0.87     17650
         Yes       0.53      0.55      0.54      4935

    accuracy                           0.80     22585
   macro avg       0.70      0.71      0.70     22585
weighted avg       0.80      0.80      0.80     22585

Random Forest Classification

             precision    recall  f1-score   support

          No       0.87      0.96      0.91     17650
         Yes       0.77      0.50      0.61      4935

    accuracy                           0.86     22585
   macro avg       0.82      0.73      0.76     22585
weighted avg       0.85      0.86      0.85     22585

K-Nearest Neighbour (KNN) Classification

              precision    recall  f1-score   support

          No       0.84      0.95      0.89     17650
         Yes       0.66      0.34      0.45      4935

    accuracy                           0.82     22585
   macro avg       0.75      0.64      0.67     22585
weighted avg       0.80      0.82      0.79     22585

Evaluation should be done on Logistic only or on all four .

Answer 1

Since you partly overfit with RF, first try to get the RF hyperparameter right. You could do a grid search like:

rf = RandomForestClassifier(...) 

param_grid = { 
    'n_estimators': [200,300],
    'max_features': [10,20,30]
}

cv = GridSearchCV(estimator=rf, param_grid=param_grid, cv= 5)
cv.fit(xtrain, ytrain)

In RandomForestClassifier max_depth and max_features are of particular interest. More trees (n_estimators) tend to be "better".

(Single) Decision Trees are usually not a good estimator.

Once you have tuned your RF properly, you could also try to "stack" KNN, logistic regression, and RF since all three of them are not too bad.

Sklearn comes with a conveniance function for stacking, namely StackingClassifier.