2
$\begingroup$

I am working on a binary classification problem. The target variable is not linearly separable, so I've decided to use LightGBM with default parameters (I only play with n_estimators on range from 10 - 100).

When I output Gain (feature importance for LightGBM) it has extremely high values on the x-axis. When I increase the number of estimators x-axis gain grows even higher. It seems to me that the model is overfitted to a single feature [1].

On the other hand, split feature importances seem to have a nice distribution without extreme values on the x-axis[2].

What is the reason for this behavior with extreme Gain values? Regularization?

Extremly high Gain of a feature

enter image description here

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Why call it overfitting? It seems to be just the most important predictor in your model. You can use column subsampling to reduce its influence resp. to shift its predictive power to other, correlated features. $\endgroup$
    – Michael M
    Dec 31, 2019 at 15:16

1 Answer 1

Reset to default
2
$\begingroup$

Its definitely not regularization since default values are 0 (check here)

n_estimators is the number of decision trees you take into bagging. These decision trees take random number of rows AND columns (again depending on the parameters) so it could be that you took some unfortunate combination which should level out with higher number of n_estimators.

Suggestion: use different criteria for feat.importance for example Permutation importance, and then compare.

Final thought, hyperparams can be irrelevant. Have a look at the effect of parameters on certain tasks, with different algorithms. Conclusion: it can be an overkill (sometimes its important!)enter image description here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.