I'm working with LightGBM on a large data set about 3M row and about 8 columns. When i started feature generation after every new feature i was measuring the RMSE of the model and if it was the same or slightly worse i was removing it so is that procedure right ? or i should do all the feature generation and then apply feature selection on them and why ?


1 Answer 1


You can add features as you create them and check if the scoring improves (and by how much). Always measure the score on test data so you can detect overfitting.

When you are done with feature engineering you can perform feature selection to reduce the number of features or understand their importances in the model.

  • $\begingroup$ but if added a feature and saw that its importance is higher than a lot of base features however, it increased the RMSE from 0.85 to 0.86 should i remove it or keep it ? $\endgroup$ May 12, 2020 at 15:15
  • $\begingroup$ You should probably keep it (because of high importance) and remove another one. Here the score goes down, probably because of overfitting. $\endgroup$
    – Rusoiba
    May 12, 2020 at 18:03
  • $\begingroup$ okay thanks a lot $\endgroup$ May 13, 2020 at 0:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.