0
$\begingroup$

Let's say I have a clear case of overfitting where my loss curves look like this (x axis are iterations): enter image description here

Now I would like to try bagging to reduce the variance, where should I stop models training? Is it in 100th iteration where model started overfitting or in 500th iteration where model has completely overfit the training data?

I tend to lean towards stopping at 100th iteration because the difference between train and test error is the lowest, but I have seen people bag decision trees which have completely overfit.

$\endgroup$
2
  • 1
    $\begingroup$ Note at iteration 100 the distance between the train and test error isn't the lowest (that was probably iteration 10), but that's the point where the validation error stops decreasing (and perhaps starts to increase again). So you're correct in your first claim that it's where the model appears to be overfitting. $\endgroup$ Sep 8, 2022 at 6:10
  • 1
    $\begingroup$ @DavidWaterworth that is a good point, I meant to say that the difference becomes bigger from that point on $\endgroup$
    – dzi
    Sep 8, 2022 at 10:16

1 Answer 1

1
$\begingroup$

You should not compare your machine learning task with others, particularly when they are overfitting their models (on other tasks).

Second, there is no mathematical rule for fixing the number $k$ of base learners (i.e., decision trees in your case). You can use grid search or cross-validation (and the like) using a validation set to optimize $k$, but if the hypothesis space is enormous you could also use random search (e.g., genetic algorithms).

Third, your approach is a good way to estimate $k$. Moreover, it also depends on the size of training and testing sets.

$\endgroup$
2
  • $\begingroup$ I am not looking for the value of k, I am just looking at where to stop the training to achieve best validation error in bagging $\endgroup$
    – dzi
    Sep 8, 2022 at 10:07
  • $\begingroup$ Which is essentially the problem of finding $k$. In particular, finding $k$ will give you, let's call it informally, the "stop point" where the validation error is reasonably low enough to avoid overfitting. $\endgroup$
    – Eduard
    Sep 8, 2022 at 11:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.