2
$\begingroup$

I am using sklearn package to make models.

I tried randomly to set some paramater to a sklearn.ensemble.RandomForestClassifier in order to have ideas about parameters to search for a future grid search.

Well, here the results of this attempt:

Accuracy :
Training: 0.9209427371993345 , Test: 0.7035561005518087

Clearly, I know it means the model is over-fitting because it's not able to generalize to new data. I know a cross-validation would be more accurate because it is possible the test-set is unfortunately too different from training-set just by random, but that's not what I am looking for in the current problematic of my topic. Besides, the two classes I try to target are imbalanced. Class 1 is more present than Class 0.

Class 0: 34% of test set/training set, Class 1: 66% of test set/training set.

So because it's imbalanced I checked about precision and recall metrics:

On test set:

+---------+-----------+--------+----------+----------+
| classes | precision | recall | F1-score | support  |
+---------+-----------+--------+----------+----------+
|       0 |      0.60 |   0.38 |     0.47 |     3326 |
|       1 |      0.73 |   0.87 |     0.80 |    6460  |
+---------+-----------+--------+----------+----------+

On training set:

+---------+-----------+--------+----------+----------+
| classes | precision | recall | F1-score | support  |
+---------+-----------+--------+----------+----------+
|       0 |      0.98 |   0.81 |     0.89 |   31265  |
|       1 |      0.91 |   0.99 |     0.95 |  59492   |
+---------+-----------+--------+----------+----------+

Then, on training set I see both classes are well predicted. Intuitively I think: if it's well predicted on the training set, it means features are good enough to split the two classes. So, it's just a matter of parameters set. But intuition is not as valuable as good experience is. So I am asking for more experienced people from this community if my intuition is wrong, and if so why it is?

UPDATES:

After checking variables, it shows distribution are the same between the training-set and testing-set through histograms.

$\endgroup$
1
$\begingroup$

A 1/3 - 2/3 repartition is not that unbalanced. Your problem shouldn't require balancing.

The train/test set partition seems to be done correctly, as it seems implied by checking data histograms. Doing that randomly is usually ok, and when it's not it will inflate your test performance with data leakage, which doesn't seems to be the case here.

Imo the problem come from your learning process, or more exactly : when does it stops ?

If you are going to explore your parameters space by hand, I would suggest to go look at the parameters of your learner : https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html, and for each one of them ask yourself what would be the impact of increasing / decreasing them on the learning capacity of your algorithm.

It may be a bit counterintuitive but you have to actcually limit the capacity of your learner, so that it does not learn too much.

Then you can do some grid search on the main parameters driving the learning / performance.

| improve this answer | |
$\endgroup$
  • $\begingroup$ This is what I envisage to do today. But I have no hope in it because by modifying the different hyper-parameters of the RF, I manage to decrease over-fitting but in the same time the generalization is still the same when I take a look in the metrics, neither improvement nor decrease. I am confused about what the problem really is. $\endgroup$ – AvyWam Jan 5 at 10:05
  • $\begingroup$ What do you mean by the generalization is the same ? $\endgroup$ – lcrmorin Jan 5 at 10:06
  • $\begingroup$ As you can see in the tables above precision and recall for the classes are 0.60/0.38 for the class 0 and 0.73/0.87 for the class 1, in test-set's metrics table. What I mean by "the generalization is the same" is I get the same values about these metrics about the test-set while over-fitting decreases. $\endgroup$ – AvyWam Jan 5 at 10:12
  • $\begingroup$ Overfitting means you learned too much on the train set. Reducing the overfitting means you will reduce the gap between performance on train and test set. Often, it will be done by mostly decreasing the performance on the train set (on which you overfit, 'over-estimating' the performance). The increase of performance on the test set is not garanteed. $\endgroup$ – lcrmorin Jan 5 at 10:32
0
$\begingroup$

You are correct, such a difference between training and test implies that the model is overfitting.

Here are some best practices to improve the process: 1. Accuracy is not a great metric for imbalanced classes and I would recommend moving to f1-score. 2. Balance the training set by over-sampling the minority class or under-sampling the majority class. 3. Retrain the model and check the new metric.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Where I have a doubt about to balance the training set with over-sampling is if it was a problem of balance the precision and recall would not be as good as it is in the training set. $\endgroup$ – AvyWam Jan 4 at 22:16
-1
$\begingroup$

Features maybe good enough but obviously you have covariate shift, or some similiar distrubancd. In other words distribution of your train and test features is different and that confuses your model, in other words it doesnt learn to differentiate on train dataset.

| improve this answer | |
$\endgroup$
  • $\begingroup$ After checking a lot, the histograms of the variables considering training-set and test-set, show the distributions are the same. $\endgroup$ – AvyWam Jan 4 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.