5
$\begingroup$

I am working on a classification problem with 4 ordinal classes to predict, labelling/predicting samples as either a number from 1-4. My training dataset has 284 features by ~40,000 samples and I am looking to explore feature correlation and variance and relate that to using a filtered feature selection method. I've been looking to learn from this guide: https://machinelearningmastery.com/feature-selection-with-real-and-categorical-data/

However, my dataset has predominantly continuous features, but it does have ~5 categorical features. How do I consider both types in my statistics and feature selection method? Is it common to partition the features and explore correlation separately (e.g. select out my categorical feature and use Chi-square, then select my continuous and use ANOVA?). Or should I be transforming either my continuous or categorical variables into the other and applying one statistical method to all features?

I aim to be systematic and explore different statistics and selection methods, but I am not sure how I should be accounting for the continuous and categorical variables together or apart. I am new to machine learning (with a biology background) so any help or resources to learn more would be appreciated.

Improve this question
$\endgroup$
3
  • $\begingroup$ Do you have a model in mind ? $\endgroup$
    – Lucas Morin
    Feb 27, 2020 at 15:37
  • $\begingroup$ I am comparing logistic regression, random forest, gradient boosting, extreme gradient boosting, and a deep neural network. Although I understand each of these might prefer a different feature selection method? Should I just filter out correlated features then leave the rest up to the models to perform their own embedded selections/weightings? $\endgroup$
    – DN1
    Feb 27, 2020 at 15:56
  • 1
    $\begingroup$ I'd say the answer is yes for logistic regression, you should aim to get a few uncorrelated variables. (start with something like 5 hand-picked variables). For more complex models, the answer is no in my opinion. If you use more complex learners that shoulc be able to learn non-linear dependencies, you shouldn't use basic linear metrics to remove some variables. This reasoning is exacerbated with more complex data (correlated / imbalanced classes). $\endgroup$
    – Lucas Morin
    Mar 2, 2020 at 9:39

1 Answer 1

Reset to default
1
$\begingroup$

I recommend using the Lasso for feature selection. I think the main advantage is that it is a multivariate method. However, it is more suited for linear models than for tree-based ones.

There are some broad theoretical and practical resources that explain why it is used. I wrote one article comparing it to univariate methods and another one comparing it to feature importance.

You can use the glmnet package in R or use the Lasso() class in scikit-learn.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.