what are the solutions for efficient feature selection in a very large feature space?

I have a classification dataset (50k observations and 10 features) on which I can't get a good result..

I want to try increasing the number of features..

I plan to automatically generate many feature options from whatever seems reasonable to me for my dataset. When I roughly calculated the number of possible signs, they turned out to be about 100 million or more ...

Naturally, such a large number of features cannot be processed at once, and 99.9% of the features will turn out to be unimportant.

My plan is this:

1. create a date set of 100/1000 features
2. train the model
3. choose features that are important from the point of view of the classification algorithm
4. save important features
5. transition to point "1" but with new features

My question is this:

1. What are the effective strategies for selecting features if there are potentially millions or more features.
2. Is my plan correct, or am I missing something?

upd_for_Dave===========================

Here is a little R code how I plan to generate features with a very simple and "small" grammar

library("gramEvol")
grammarDef <- CreateGrammar(list(
expr  = grule(op(expr, expr), func(expr), var),
func  = grule(sin, cos, log, sqrt),
op    = grule("+","-","*","/"),
var   = grule(var, var^n),
n     = gvrule(1:4),
var   = grule(x1,x2)))


Here are the potential features that this grammar generates

gramEvol::GrammarRandomExpression(grammarDef,numExpr = 10)

[[1]]
expression(x2)

[[2]]
expression(x2)

[[3]]
expression(cos(sqrt(x1 + sqrt(x1)) + sqrt(x1)))

[[4]]
expression(cos(sqrt(sin(sqrt(cos(log(sin(log(x2))) * cos(sqrt(cos(x1)))))))) - cos(x2 * x2))

[[5]]
expression(log(x1))

[[6]]
expression(x2)

[[7]]
expression(sqrt(cos(x2 * (x1 - sqrt(log((sqrt(x1) + x1)/x2/x2)/(sin(x1) * x2))))))

[[8]]
expression(cos(x1))

[[9]]
expression(x2)

[[10]]
expression(cos(x2))


So many unique combinations

summary(grammarDef)
No. of Unique Expressions:   3.993138e+15


And this is only with this simple grammar, but you can do not only mathematical expressions, but code, text .. anything .. But I think this is all beyond the scope of the question.

• How are you generating those millions of features?
– Dave
Nov 15, 2022 at 12:35
• As a "feature generator" I'm going to use a symbolic regression/genetic progmaming library.. With this algorithm, I can essentially generate code that in turn generates features of any type
– mr.T
Nov 15, 2022 at 13:42
• @Dave see my update
– mr.T
Nov 15, 2022 at 14:07
• What do you do once you get these features? For instance, do you run a linear regression on them?
– Dave
Nov 15, 2022 at 14:09
• I was thinking of using XGboost to immediately extract important features
– mr.T
Nov 15, 2022 at 14:13