I am working on a classification problem with 4 ordinal classes to predict, labelling/predicting samples as either a number from 1-4. My training dataset has 284 features by ~40,000 samples and I am looking to explore feature correlation and variance and relate that to using a filtered feature selection method. I've been looking to learn from this guide: https://machinelearningmastery.com/feature-selection-with-real-and-categorical-data/

However, my dataset has predominantly continuous features, but it does have ~5 categorical features. How do I consider both types in my statistics and feature selection method? Is it common to partition the features and explore correlation separately (e.g. select out my categorical feature and use Chi-square, then select my continuous and use ANOVA?). Or should I be transforming either my continuous or categorical variables into the other and applying one statistical method to all features?

I aim to be systematic and explore different statistics and selection methods, but I am not sure how I should be accounting for the continuous and categorical variables together or apart. I am new to machine learning (with a biology background) so any help or resources to learn more would be appreciated.

  • $\begingroup$ Do you have a model in mind ? $\endgroup$ – lcrmorin Feb 27 '20 at 15:37
  • $\begingroup$ I am comparing logistic regression, random forest, gradient boosting, extreme gradient boosting, and a deep neural network. Although I understand each of these might prefer a different feature selection method? Should I just filter out correlated features then leave the rest up to the models to perform their own embedded selections/weightings? $\endgroup$ – DN1 Feb 27 '20 at 15:56
  • 1
    $\begingroup$ I'd say the answer is yes for logistic regression, you should aim to get a few uncorrelated variables. (start with something like 5 hand-picked variables). For more complex models, the answer is no in my opinion. If you use more complex learners that shoulc be able to learn non-linear dependencies, you shouldn't use basic linear metrics to remove some variables. This reasoning is exacerbated with more complex data (correlated / imbalanced classes). $\endgroup$ – lcrmorin Mar 2 '20 at 9:39

I recommend using the Lasso for feature selection. I think the main advantage is that it is a multivariate method. However, it is more suited for linear models than for tree-based ones.

There are some broad theoretical and practical resources that explain why it is used. I wrote one article comparing it to univariate methods and another one comparing it to feature importance.

You can use the glmnet package in R or use the Lasso() class in scikit-learn.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.