I have a dataset with a large number of potential features (>100) and I am interested in finding a relatively small subset of these (maybe on the order of 5, or 20) features which is best suited to solving a specific type of problem. What are some good ways to evaluate which columns of my dataset are best suited to use as inputs to solve the problem, and which I should discard? (The nature of the problem is approximating the inverse of some complicated mathematical functions).

  • 2
    $\begingroup$ Variable selection is highly problematic and overrated. In light of these facts, why do you feel the need to select particular features? $\endgroup$
    – Dave
    Jun 22 at 16:08
  • $\begingroup$ The nature of the problem is generative, so having to specify points in say 5 dimensions instead of 100 would be very helpful. More broadly part of the goal of the project is to find a 'basis' in which every input vector uniquely identifies a single output vector, so it makes sense naively to have n constraints for n unknowns. $\endgroup$
    – nighthawk
    Jun 22 at 16:48
  • 2
    $\begingroup$ I think that the answer depends a lot on which 'specific problem' you want to solve. For example, if the problem involves predicting a specific target variable, there are methods to select the most relevant features. But the set of features is different for different target variables. $\endgroup$
    – Erwan
    Jun 23 at 9:14
  • 1
    $\begingroup$ Essentially I have a dataset where a large number of potential features have been calculated via multivariable functions from a comparatively small number of inputs, through third-party software. I want to learn the inverse of these functions for some subset of these features. So I think, to borrow your phrasing, the goal would be to predict a vector of multiple specific target variables. $\endgroup$
    – nighthawk
    Jun 23 at 16:23

1 Answer 1


IF "the goal would be to predict a vector of multiple specific target variables" - look for Multivariable Multivariate Analyse.

First, do Exploratory Data Analysis -- to notice real Dependencies exist -- you need to select some informative predictors at a glance (e.g. with df pair_plots taking into considerarion Covariations & Correlations) & from boxplots can get rather info about features themselves

Then you should take away Multicollinearity (e.g. see VIF factor or correlation matrix, or do dimencionality reduction with a help of orthogonal rotation)

You can also remove Noise from your initial data by using filter - EMA e.g. (Exponential Moving Average)

You can remove or trim outliers (if they are real errors) at any stage of your analysis - to get Normal distribution, or use RobustScaler (to take outliers into consideration)

Can make log-transformations for skewed data - to get Normal bell-curve of Targets (Normality of DV provides trustfull results)... IVs (independent variables) don't need obligatory to be normal. But scaling for IVs is obligatory to avoid variance inflation by certain features

And, of course, grouping variables to ranges also serves to the aim of generalization

In any way, you should always remember the formula "garbage in - garbage out". Either you will filter for garbage in your data preprocessing or in model - depends on you. To my consideration - it is easier & serves to better performance if doing it in pre-modelling stage.

But in any case, DataScience's goal is to investigate/explore data for some dependencies of DVs from latent factors (predictors) & it's up to you - what & how to explore your data, what design of experiments to follow in order to substract meaningful predictors for certain targets - while comparing your experimental results with ANOVA methods (or others for non-normally distributed variables); but ANOVA can investigate only target variable under different conditions & interactions between them, so - exploration of each predictor by its own - is your responsibility (to proove whether the cause-effect-dependancy mathematically/statistically obviously exists && if it is logical by its nature, because not always Correlation means Causation). ==> To my opinion: any ML algo do not lead to statistical significance. And at the same way, you can trust to statistical significance only if you apply correct stat.methods (all of them have their own assumptions and goals they are used for). But ML results are not statistically approved (just averaged with maximum likelihood method by any of gradient type usage), -- by the way, stat is not the goal of ML algos

P.S. of course, at the end you should check your regression for autocorrelation of residuals to be sure it is absent, or you will need to change your model or input data

  • $\begingroup$ if just ML using - you can see - Feature Importance - that says e.g., that "Model like RandomForest have built-in feature importance" - but be sure you understand the algo: its aim, its adequecy for concrete distribution, and nature of features it is assigned to be used for (categorical or numerical)... It's hard for me to blindly believe without knowing the algorithm and especially its assumptions $\endgroup$
    – JeeyCi
    Jun 26 at 6:06
  • $\begingroup$ smth like this branch - Feature importances with a forest of trees $\endgroup$
    – JeeyCi
    Jun 26 at 6:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.