Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
when you create the new feature for data analysis
for linear regression, it is clear that the feature has to be linear with other features is better
but for xgboost what is the guideline to make a new feature.
1. what is the guideline to new features for xgboost
2. when xgboost split out the feature importances. is it better just drop the non-important features?
3. if I want to know about this related field what should I study please let me know the link and books or other resorucees
1) There are no guidelines for new features for xgboost, or even linear regression. It is not necessarily true that features have to be linearly related to the outcome in a linear regression, one can use transformations and interactions to capture non-linear relationships. The problem? In a linear regression we have to manually model the non-linearity, which is a pain to find if you have more than just a handful of features and not much prior information. Xgboost (hopefully) finds these non-linear relationships automatically.
The "guidelines" for new features, regardless of your model, is really driven by the actual problem at hand. That is, what knowledge do you have about the possible relationship between a variable X and Y, and furthermore, are there better ways to express the relationship using the current information that you do know? Here is an example; suppose I have a problem where I wish to predict the probability of some customer defaulting on a loan I provide to them. I am given their past payment history, provided as a series of payments (if made). I can use external knowledge from my background in finance to know that there is an explicit formula to calculate the given interest rate in the past, given a series of payments at known intervals. I can then use this calculated interest rate as a variable to get a better representation for the relative risk of a client (as opposed to just aggregating the payments themselves) since I know that riskier people get higher interest rates due to a higher implicit probability of defaulting. Another example: in insurance, people in my country typically get into fewer accidents in the winter than the summer. Perhaps, if I were trying to predict accident occurrence over time I would strive to include some seasonality indicator variables.
I have read some guidelines in this book here by Kuhn and Johnson regarding interaction features, namely that more important features are generally more likely to be better candidates to include as interactions (product of two or more predictors are interactions, in case you were wondering).
This process is commonly referred to as feature engineering, where we essentially manipulate our current data such that the model can learn easier.
2) Maybe. For some problems yes, for other problems, no. The curse of dimensionality is real and definitely can lead to overfitting. One can use the feature importances from xgboost to drop variables and improve their model but often you can overfit to the training set when doing this and you may actually make your model worse than if you had done nothing at all. Hence, it is incredibly important that you wrap this process in a cross validation scheme to see if the feature selection process actually improves your model. Two popular algorithms that make use of feature importance scores for variable selection are recursive feature elimination and permutation methods (so called "null importances").
3) This topic is unfortunately not as well covered in literature though feature selection, extraction, and engineering are very fruitful topics being researched right now. That being said, this aforementioned book might be of use to you (current a work in progress, so it might not be free forever):
