I noticed that developing ml models a very important step in feature engineering is adding new features that can explain better the target variable. Recently I experienced a situation where by adding some features the model's performance improved, while adding five more features the performance worsened. So, my question is: is the feature engineering process more an experimentation phase where I need to try different scalers, features and encoding methods? If I am wrong please let me know.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I would recomend reading (Feature Engineering and Selection: A Practical Approach for Predictive Models).
Intuitively, I would say that adding relevant and informative features to a model can help it capture more patterns in the data, leading to improved performance. For example, in a classification problem, adding features that are strongly correlated (and not by chance) with the target variable can increase the accuracy of the model. If you get to explain a significant portion of the variance in the target variable, you get to improve the model's predictive power.
However, adding irrelevant or redundant features can lead to a decrease in performance. This is because such features can introduce noise (obscuring the underlying pattern) and complexity to the model (making variables adjust one another), making it harder to learn the underlying patterns in the data. Additionally, if the added features are highly correlated with existing features, it can lead to multicollinearity, which can also reduce the model's accuracy. You can see the coefficients of a logistic regression, they get huge when they are collinear.
I would add that the number of variables will depend on the amount of data you have for each of the states of the predictor variables you are using. You can delve further into this by reading about what Frank Harrel has written about the limiting sample size.
