I need to perform a feature selection on my dataset. My dataset is an imbalanced dataset where the class of interest is the minority class. Therefore, recall and F2 measures are two important metrics to consider to evaluate the performance of the prediction. I used two methods to select features.
Firstly, I partitioned the data into training and test set and then apply
Info gain attribute selection method to select top 30 attributes. I kept only 30 attributes in the training set and discard all other attributes. I kept those 30 attributes for test set as well and use machine learning algorithms for prediction.
The second method, I perform attribute selection in the whole dataset and kept top 30 attributes and discard all other attributes from the dataset. Afterward, I partitioned the data into training and test set.
For logistic regression I got recall for the 1st method = 0.802 whereas I got recall for the 2nd method is 0.830.
Now I am confused about which method I need to follow. 1st method or the 2nd method? Moreover, I like to know, which one is the actual process for machine learning while training the model. Is there any sequence exists between feature selection and data partition?