Given the anonymized dataset of features below, where:

  1. "code" is a categorical variable.
  2. "x1" and "x2" are continuous variables.
  3. "x3" and "x4" are extracted features. They are the mean values of "x1" and "x2" respectively for each individual code.
       code  x1  x2  x3  x4
    0   100   1   2   2   4
    1   100   2   4   2   4
    2   100   3   6   2   4
    3   200   4   8   5  10
    4   200   5  10   5  10
    5   200   6  12   5  10
    6   300   7  14   8  16
    7   300   8  16   8  16
    8   300   9  18   8  16

Looking at the columns, for each code, x3 and x4 features have similar values - when x3 is 2 or 5 or 8, the code would be 100 or 200 or 300 respectively, and when x4 is 4 or 10 or 16, the code would be 100 or 200 or 300 respectively.

Intuitively, leaving these columns as they are without dropping any would lead to redundant features while training a model. My question is how true is this my hypothesis? I'm not so confident about it. Does it really matter when training a model? Does it depend on the model type (tree based or otherwise)?


1) Those extracted features are "combination" of original features. Yes, it's additional information. And yes, it could be redundant, but I'd check that after creating those features by doing standard feature selection operations. Here's a great guide from Kaggle. At the same time, it might be a very good feature engineering decision - completely depends on a dataset.

2) Yes, it matters - redundant information can negatively influence model performance. But you find out to understand whether it's redundant or not. The minimum bad thing you can face - is how much time does training takes (more features - more computation resources). The more serious thing is overfitting.

3) Yes, and it depends on how your model is able to process categorical features. If it cannot do anything specific with them, you should remove the original categorical features. XGboost or CatBoost (those are tree-based both) are able to process categorical features from the box. Generally, tree-based models work good with a lot of features, from my experience. And linear models require right feature selection.

Also, here's a link to a great example on Target Feature encoding, it might be also useful, however, you must be very attentive to train/test splits and overfitting using it.

|improve this answer|||||
  • 1
    $\begingroup$ Your answers and the concept of target encoding really helps. I've come across target encoding before, now I understand the concept and how it relates to my problem. $\endgroup$ – user9269433 Nov 5 '19 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.