I am working on a binary classification problem. The target variable is not linearly separable, so I've decided to use LightGBM with default parameters (I only play with n_estimators on range from 10 - 100).

When I output Gain (feature importance for LightGBM) it has extremely high values on the x-axis. When I increase the number of estimators x-axis gain grows even higher. It seems to me that the model is overfitted to a single feature [1].

On the other hand, split feature importances seem to have a nice distribution without extreme values on the x-axis[2].

What is the reason for this behavior with extreme Gain values? Regularization?

Extremly high Gain of a feature

enter image description here

  • 2
    $\begingroup$ Why call it overfitting? It seems to be just the most important predictor in your model. You can use column subsampling to reduce its influence resp. to shift its predictive power to other, correlated features. $\endgroup$ – Michael M Dec 31 '19 at 15:16

Its definitely not regularization since default values are 0 (check here)

n_estimators is the number of decision trees you take into bagging. These decision trees take random number of rows AND columns (again depending on the parameters) so it could be that you took some unfortunate combination which should level out with higher number of n_estimators.

Suggestion: use different criteria for feat.importance for example Permutation importance, and then compare.

Final thought, hyperparams can be irrelevant. Have a look at the effect of parameters on certain tasks, with different algorithms. Conclusion: it can be an overkill (sometimes its important!)enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.