In between data partition and feature selection which one need to perform 1st

Question

I need to perform a feature selection on my dataset. My dataset is an imbalanced dataset where the class of interest is the minority class. Therefore, recall and F2 measures are two important metrics to consider to evaluate the performance of the prediction. I used two methods to select features.

Firstly, I partitioned the data into training and test set and then apply Info gain attribute selection method to select top 30 attributes. I kept only 30 attributes in the training set and discard all other attributes. I kept those 30 attributes for test set as well and use machine learning algorithms for prediction.

The second method, I perform attribute selection in the whole dataset and kept top 30 attributes and discard all other attributes from the dataset. Afterward, I partitioned the data into training and test set.

For logistic regression I got recall for the 1st method = 0.802 whereas I got recall for the 2nd method is 0.830.

Now I am confused about which method I need to follow. 1st method or the 2nd method? Moreover, I like to know, which one is the actual process for machine learning while training the model. Is there any sequence exists between feature selection and data partition?

Answer 1

You are using "Info Gain" as the attribute to select your features. If you aren't certain whether that is accurate enough, I would try basic feature importance. If you are using a tree model, it should be built in. If you are using a Neural Network, use permutation feature importance. That way you can plot the importance, and decide to use the top N features.

If you are interested in a particular class, you might want to calculate the feature importance of only that class' data points after training the model on the dataset as a whole. That way you can see which features are most correlated with the class of interest