I'm working with LightGBM on a large data set about 3M row and about 8 columns. When i started feature generation after every new feature i was measuring the RMSE of the model and if it was the same or slightly worse i was removing it so is that procedure right ? or i should do all the feature generation and then apply feature selection on them and why ?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Cross-posted at stats.stackexchange.com/q/465894/232706 $\endgroup$– Ben Reiniger ♦Commented May 11, 2020 at 23:23
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
You can add features as you create them and check if the scoring improves (and by how much). Always measure the score on test data so you can detect overfitting.
When you are done with feature engineering you can perform feature selection to reduce the number of features or understand their importances in the model.
-
$\begingroup$ but if added a feature and saw that its importance is higher than a lot of base features however, it increased the RMSE from
0.85to0.86should i remove it or keep it ? $\endgroup$ Commented May 12, 2020 at 15:15 -
$\begingroup$ You should probably keep it (because of high importance) and remove another one. Here the score goes down, probably because of overfitting. $\endgroup$– RusoibaCommented May 12, 2020 at 18:03