-1
$\begingroup$

Im building a sentiment classification model using RandomForestClassifier. I got the training accuracy of 99.65 & cross-validation( RepeatedStratifiedKFold-5 folds) accuracy of 97.29. I used f1 score for metrics. The dataset size is 5184 samples. The dataset is imbalanced so i'm using class_weight hyper-parameter as 'balanced'. I have done hyper parameter tuning also. Following are the parameters i tuned -

estimator = RandomForestClassifier(random_state=42, class_weight='balanced', n_estimators=850, min_sample_split=4, max_depth=None, min_samples_leaf=1, max_features='sqrt')

Im thinking the model is overfitting. Im also wondering is this issue caused because of the class imbalance?

Any immediate help on this is much appreciated.

$\endgroup$
2
  • $\begingroup$ I'd say going down from 99.65 to 97.29 is not a strong sign of overfitting, maybe a bit. How many features? A low ratio instances/features is likely to cause overfitting. $\endgroup$
    – Erwan
    Commented Sep 7, 2020 at 11:16
  • $\begingroup$ 5184 rows and 30884 features $\endgroup$
    – emily
    Commented Sep 9, 2020 at 4:05

1 Answer 1

2
$\begingroup$

There's quite a lot of features for the number of instances, so it's indeed likely that there's some overfitting happening.

I'd suggest these options:

  • Forcing the decision trees to be less complex by setting the max_depth parameter to a low value, maybe around 3 or 4. Run the experiment with a range of values (e.g. from 3 to 10) and observe the changes in performance (preferably use a validation set, so that when the best parameter is found you can do the final evaluation on a different test set).
  • Reducing the number of features: remove rare words (i.e. those which appear less than $N$ times) and/or use some feature selection method.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.