1
$\begingroup$

i am testing gradient boosting regressor from sklearn for time series prediction on noisy data (currency markets).

https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html

surprisingly, the the gradient boosting regressor achieves very high accuracy on the training data - surprising because the data is so noisy. however, it performs poorly on the test set.

this is clearly a case of overfitting, so i'm wondering what parameters i can change to regularize the gradient boosting regressor.

so far i've tried max_depth, reducing it to 1 (from the default of 3). this seems to work pretty well in increasing accuracy on the validation set.

does anyone know what other parameters i could tweak, to improve performance on the validation/test set? thanks

$\endgroup$
  • $\begingroup$ Are you forecasting future values using your gradient boosting model (i.e. extrapolation?) Note that you do not have independent observations here (correlation with time) and gradient boosting models have difficulty extrapolating beyond what is observed in the training set. In particular, if you have a trend in your time series you need to explicitly model this as a feature fed to the gradient boosting machine, perhaps as a linear model. $\endgroup$ – aranglol Dec 2 '19 at 16:50
1
$\begingroup$

The hyper parameters that you could tune in any boosting technique are:

  1. Depth of each tree: As you rightly pointed out this is very important because each tree in boosting technique learns from the errors of the previous trees. Hence underfitting the initial trees ensure that the later trees learn actual patterns and not noise.

  2. Number of trees: this is kind of intuitive from previous point as the number of trees increase the learnable signal decreases and hence the ideal number of trees is more than underfitting trees and less than overfitted trees.

  3. Learning rate: this parameter gives weights to previous trees according to a value between 0 and 1. Lower learning rates give lesser importance to previous trees. Higher weights lead to faster steps towards optimization. Lower weights typically lead to global optimum. But lower learning rates need more trees to learn the function.

4.Sub sample: if the value is less than 1 a subset of variables is used to build the tree making it robust and learn signal from more variables. This variable reduces overfitting by not fitting only 1 variable but a group of variables.

These variables if tuned correctly are sufficient to reduce overfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.