How to perform feature selection with Categorical Variables and Continuous Target, provided that data is not normally distributed?

Question

I am trying a use multi linear regression model to predict the salaries of employees. I have a total of 88 dependent features from which 19 are categorical and the rest are continuous. I have managed to reduced the number of continuous features from 69 to 41. Now I am trying to reduce the number of categorical features, but since my data is not normally distributed I cannot use a t-test or ANOVA. Which other tests can I use to see if the features are significant to predict the target?

Answer 1

Be careful with feature selection! Do not rely solely on feature selection techniques. They might be misleading sometimes. Here is the process I usually follow:

1.) The very first thing to do is build a baseline model where you consider all the features and record the performance. This will give you a baseline score to compare with. (Do not perform hyperparameter tuning here!)

2.) Now you perform feature engineering, where you see if you can combine multiple feature into one single one. For example you have 3 features as date, month and year of sale of a car. You can combine all 3 of them into a single feature age. This will reduce the dimension of your dataset.

3.) Here you try to remove any outliers/nonsensical values from features. For example in the case of predicting the price of car, you have year of car as 1900. This is a nonsensical value and won't help the model. You can safely remove it. (be carefull how you deal with outliers as removing them is not the only solution but thats a whole another topic in itself!)

4.) Now you can perform feature selection. There a quite a few techniques you can use like filter based, wrapper based and hybrid techniques. But don't just blindly use of these as they might be misleading. Instead use subject matter expertise first to remove any redundant features (which is what I usually do).

Applying all the above things will usually result in the removal of redundant features. If not only then go for feature selection techniques mentioned in point 4.).

Hope it helps you!

Answer 2

If I understand your question correctly, you are asking how to reduce categorical features in a dataset. If yes, then a few of the approaches I can think of are:

1. Iterative Process - Build a model with all numerical features and one categorical feature then evaluate your improvement of the model by whatever metrics you are using and then add other categorical features and so on. So if you have N categorical features you will be building N+1 models.

2. chi square test of predictor and target variables.

3. (what I use) Build a model with all the available features and measure its performance and then use the feature importance functionality of that model to determine which features are important. In the case of linear regression, the higher the value of the coefficient the better the feature. Alternatively, you can use L1 regularization to check for non zero features. Do check for multi-collinearity before considering feature importance in linear regression.