I have a dataset with a large number of potential features (>100) and I am interested in finding a relatively small subset of these (maybe on the order of 5, or 20) features which is best suited to solving a specific type of problem. What are some good ways to evaluate which columns of my dataset are best suited to use as inputs to solve the problem, and which I should discard? (The nature of the problem is approximating the inverse of some complicated mathematical functions).
IF "the goal would be to predict a vector of multiple specific target variables" - look for Multivariable Multivariate Analyse.
First, do Exploratory Data Analysis -- to notice real Dependencies exist -- you need to select some informative predictors at a glance (e.g. with df pair_plots taking into considerarion Covariations & Correlations) & from boxplots can get rather info about features themselves
Then you should take away Multicollinearity (e.g. see VIF factor or correlation matrix, or do dimencionality reduction with a help of orthogonal rotation)
You can also remove Noise from your initial data by using filter - EMA e.g. (Exponential Moving Average)
You can remove or trim outliers (if they are real errors) at any stage of your analysis - to get Normal distribution, or use RobustScaler (to take outliers into consideration)
Can make log-transformations for skewed data - to get Normal bell-curve of Targets (Normality of DV provides trustfull results)... IVs (independent variables) don't need obligatory to be normal. But scaling for IVs is obligatory to avoid variance inflation by certain features
And, of course, grouping variables to ranges also serves to the aim of generalization
In any way, you should always remember the formula "garbage in - garbage out". Either you will filter for garbage in your data preprocessing or in model - depends on you. To my consideration - it is easier & serves to better performance if doing it in pre-modelling stage.
But in any case, DataScience's goal is to investigate/explore data for some dependencies of DVs from latent factors (predictors) & it's up to you - what & how to explore your data, what design of experiments to follow in order to substract meaningful predictors for certain targets - while comparing your experimental results with ANOVA methods (or others for non-normally distributed variables); but ANOVA can investigate only target variable under different conditions & interactions between them, so - exploration of each predictor by its own - is your responsibility (to proove whether the cause-effect-dependancy mathematically/statistically obviously exists && if it is logical by its nature, because not always Correlation means Causation). ==> To my opinion: any ML algo do not lead to statistical significance. And at the same way, you can trust to statistical significance only if you apply correct stat.methods (all of them have their own assumptions and goals they are used for). But ML results are not statistically approved (just averaged with maximum likelihood method by any of gradient type usage), -- by the way, stat is not the goal of ML algos
P.S. of course, at the end you should check your regression for autocorrelation of residuals to be sure it is absent, or you will need to change your model or input data