I have a classification dataset (50k observations and 10 features) on which I can't get a good result..

I want to try increasing the number of features..

I plan to automatically generate many feature options from whatever seems reasonable to me for my dataset. When I roughly calculated the number of possible signs, they turned out to be about 100 million or more ...

Naturally, such a large number of features cannot be processed at once, and 99.9% of the features will turn out to be unimportant.

My plan is this:

  1. create a date set of 100/1000 features
  2. train the model
  3. choose features that are important from the point of view of the classification algorithm
  4. save important features
  5. transition to point "1" but with new features

My question is this:

  1. What are the effective strategies for selecting features if there are potentially millions or more features.
  2. Is my plan correct, or am I missing something?


Here is a little R code how I plan to generate features with a very simple and "small" grammar

grammarDef <- CreateGrammar(list(
  expr  = grule(op(expr, expr), func(expr), var),
  func  = grule(sin, cos, log, sqrt),
  op    = grule("+","-","*","/"), 
  var   = grule(var, var^n),
  n     = gvrule(1:4),
  var   = grule(x1,x2)))

Here are the potential features that this grammar generates

gramEvol::GrammarRandomExpression(grammarDef,numExpr = 10)



expression(cos(sqrt(x1 + sqrt(x1)) + sqrt(x1)))

expression(cos(sqrt(sin(sqrt(cos(log(sin(log(x2))) * cos(sqrt(cos(x1)))))))) - cos(x2 * x2))



expression(sqrt(cos(x2 * (x1 - sqrt(log((sqrt(x1) + x1)/x2/x2)/(sin(x1) * x2))))))




So many unique combinations

No. of Unique Expressions:   3.993138e+15

And this is only with this simple grammar, but you can do not only mathematical expressions, but code, text .. anything .. But I think this is all beyond the scope of the question.

  • $\begingroup$ How are you generating those millions of features? $\endgroup$
    – Dave
    Nov 15, 2022 at 12:35
  • $\begingroup$ As a "feature generator" I'm going to use a symbolic regression/genetic progmaming library.. With this algorithm, I can essentially generate code that in turn generates features of any type $\endgroup$
    – mr.T
    Nov 15, 2022 at 13:42
  • $\begingroup$ @Dave see my update $\endgroup$
    – mr.T
    Nov 15, 2022 at 14:07
  • $\begingroup$ What do you do once you get these features? For instance, do you run a linear regression on them? $\endgroup$
    – Dave
    Nov 15, 2022 at 14:09
  • $\begingroup$ I was thinking of using XGboost to immediately extract important features $\endgroup$
    – mr.T
    Nov 15, 2022 at 14:13

1 Answer 1


Your version is good enough, but features based on 1 feature can also be used. I use Weight of Evidence (WOE) and Information Value (IV) when there are many features. I would bin the variable, calculate the IV, throw out the obviously bad ones (iv < 0.03 for example) and make a feature / iv table. I also have some heuristic rule: I will add features to the model as long as they add at least 0.01 roc_auc_score. If the 1-variable model has abs(roc_auc_score) < 0.501 then I remove it from the sample.

  • $\begingroup$ Thanks Andrew! I have a few questions if you don't mind. 1) Did I understand correctly? I need to train a single feature model with good WOE/IV and if it gives a positive ROC AUC then add it to the overall big model? 2) The correlation of signs is not taken into account in any way, right? 3) I heard that WOE is suitable for logistic regression and cannot catch non-linear relationships, is that so? $\endgroup$
    – mr.T
    Nov 16, 2022 at 7:03
  • $\begingroup$ Not quite. I propose 2 one-dimensional criteria to remove obviously weak functions. 1 small iv<0.02, 2 small quality metric (roc_auc for example). Yes, indeed, iv is suitable for linear models, but my heuristic is that too small iv will not give an increase in the quality metric in a nonlinear model. $\endgroup$
    – Andrew
    Nov 17, 2022 at 10:06
  • $\begingroup$ Since the number of features is too large to calculate "honestly" the importance of certain features, I suggest throwing out the features according to the principle: if the feature itself does not display the target variable in any way, then it is not needed in the final model. Since you seem to generate all sorts of combinations anyway, then you don't need weak features that, in certain rare cases, can give an increase in quality metrics in non-linear combinations. After all, this information will be taken into account in another good sign. $\endgroup$
    – Andrew
    Nov 17, 2022 at 10:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.