10
$\begingroup$

I have a dataset with 20 variables and ~50K observations, I created several new features using those 20 variables.

I compare the results of a GBM model (using python xgboost and light GBM) and I found that it doesn't matter what are the hyper-parameters of the model the 'thinner' version leads to better results (AUC) even that all 20 original variables are included in the wider version.

when I compare the same using Lasso model - the wider version is better (~1% higher) as expected.

I guess it can be related to to the randomness in the GBM but I was surprised to see that GBM doesn't fix it along the way.

Any explanation of the phenomena will be appreciated.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
7
$\begingroup$

To put it shortly, xgboost tries to fix it and although it is very good in getting rid of overfitting, it is not perfect.

Adding new features is not always beneficial, because you increase the dimension of your search space and thus make the problem harder. In your particular case the increased complexity overweight the added value from extra features.

I understand you‘ve tested quite wide range of hyper-parameters and enough combinations. If you apply regularisation via colsample_bytree and/or colsample_bylevel it might happen that at some stage the randomly chosen columns (features) are less informative than your original features and the algorithm is forced to use these features for further splits. Does it make sense to you?

The number of added features might be crucial, if it is too high relative to original 20 the new features become just too dominant. For example, this might happen if one adds some nominal features with high cardinality, which are then dummy-encoded.

In order to improve the results with wider data you might want to play with the parameters controlling the early stopping and stop the fitting even earlier.

Edit

The rationale for my last suggestion: I assumed that worsened performance on wider dataset is due to overfitting, i.e. significantly worse performance on test dataset than on training data. Early stopping should prevent / control the overfitting, but it seems it didn’t work so good in your case and thus should be improved.

You can and should test different combination of new features, but trustworthy quality metric is crucial for the model choice. If you significantly overfit, the performance on training data (or even cross-validation) will not help you to choose the model (or feature combination) with good performance on test data set.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I agree with all your answer, this is exactly what I thought that's happening. I'm not using OHE, I'm using target encoding instead bu't it's already part of the 20 variables. can you elaborate of why do you believe that early stopping might help? as I see it the problem is to find the right subset of variables (in some way like finding the best hyper-parameters) $\endgroup$
    – Yaron
    Commented Dec 9, 2017 at 18:25
  • $\begingroup$ My last edit should hopefully address your question. Btw, (the incorrectly performed) target encoding might lead to data leakage and lead to biased results. If the cardinality of your nominal features permits, you might test the performance of OHE. $\endgroup$
    – aivanov
    Commented Dec 10, 2017 at 16:07
3
$\begingroup$

Welcome to the site!

If I understand your question correctly you want to know why a model would perform worse when a new feature is added?

So every time you do feature engineering (add new columns, derive columns, standardize the data, normalize the data, etc) there is always a flip side of the coin. If you add some features and if those features explain something about the target variable then it would aid in increasing the accuracy; on the other hand, if the feature doesn't have much relation to the target variable then it doesn't aid in increasing the accuracy.

Now before going to modeling you can go through a couple of things like (assuming that you have done all these things):

  1. Eliminating Unnecessary Features using Business Understanding
  2. Removing Outliers
  3. Imputing Missing Data (XGBoost is not prone to missing values)
  4. Standardizing data
  5. Correlation analysis with the target variable and within the variables too, eliminate if there is high correlation between variables (as you need independent variables), eliminate the variables which are not much related to the target variable.

Once the above steps are done you can get a model which explains the data well. To improve the accuracy you need to do more feature engineering and try to find if there are some external factors which might effect your model.

There are many reasons why a model doesn't work well. Some of the above stated might be the reason why your model is not performing well in your scenario. Do have a closer look on the data -- this might give you a better idea. This is a top level view.

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ hi, thanks for your reply, but I think it doesn't answer the problem which is why adding extra variables deteriorate the model performance. beside that: 1. the goal is to automate as much as i can the process so deleting using 'business understanding' is less relevant. 2. regarding you 5th comment: keeping only the correlated variables can missed a lot because less correlated variables are adding to the model as well (try for example re-running GBM model with only the 10 highest important and you'll get worse model) $\endgroup$
    – Yaron
    Commented Dec 7, 2017 at 7:57
  • $\begingroup$ So you know the process of things which you need to follow before doing modelling right? Yes you can automate the process for the next iterations where you feel that these fields are important but from the start itself you cannot except everything to be perfect. See generally in Data Science field there nothing like benchmark, when we go for correlation with the target variable we make sure that the correlation is around 0.5 above but this is not hard and fast rule. If you keep variables less than that then you model wouldn't give the results which you are expecting. $\endgroup$
    – Toros91
    Commented Dec 7, 2017 at 8:01
  • 1
    $\begingroup$ How could you say that model would worsen if you take 10 important variables, Let us consider a scenario where you have 15 variables and 10 variables explains the most then keeping the other 5 variables would worsen the model as the model would generally give importance to all the variables, since the last 5 variables are not explaining much about the target variable. What ever i'm saying here is out of my experience but I'm 100% correct but there might be some exceptional cases. Yours might also be one of them. $\endgroup$
    – Toros91
    Commented Dec 7, 2017 at 8:05
  • $\begingroup$ I recommend you to test what you offer, develop GBM model with 20 variables -> keep only the ten most important -> refit the model -> test results on test sample. you'll see worse model. the reason is that those variables are adding to the model but not as the others $\endgroup$
    – Yaron
    Commented Dec 7, 2017 at 8:13
  • $\begingroup$ give me your sample data, will try implementing and update you back asap. I might be wrong, would love to test it. $\endgroup$
    – Toros91
    Commented Dec 7, 2017 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.