I need to perform a feature selection on my dataset. My dataset is an imbalanced dataset where the class of interest is the minority class. Therefore, recall and F2 measures are two important metrics to consider to evaluate the performance of the prediction. I used two methods to select features.

Firstly, I partitioned the data into training and test set and then apply Info gain attribute selection method to select top 30 attributes. I kept only 30 attributes in the training set and discard all other attributes. I kept those 30 attributes for test set as well and use machine learning algorithms for prediction.

The second method, I perform attribute selection in the whole dataset and kept top 30 attributes and discard all other attributes from the dataset. Afterward, I partitioned the data into training and test set.

For logistic regression I got recall for the 1st method = 0.802 whereas I got recall for the 2nd method is 0.830.

Now I am confused about which method I need to follow. 1st method or the 2nd method? Moreover, I like to know, which one is the actual process for machine learning while training the model. Is there any sequence exists between feature selection and data partition?

  • $\begingroup$ 2nd method is wrong, and it is known to have led lots of people astray; we never involve our test set in any part of the process (including feature selection). See own answers in SO: stackoverflow.com/questions/56308116/… $\endgroup$
    – desertnaut
    Oct 23, 2022 at 23:35

1 Answer 1


You are using "Info Gain" as the attribute to select your features. If you aren't certain whether that is accurate enough, I would try basic feature importance. If you are using a tree model, it should be built in. If you are using a Neural Network, use permutation feature importance. That way you can plot the importance, and decide to use the top N features.

If you are interested in a particular class, you might want to calculate the feature importance of only that class' data points after training the model on the dataset as a whole. That way you can see which features are most correlated with the class of interest

  • 1
    $\begingroup$ I am afraid this does not answer the question; please re-read it closely. $\endgroup$
    – desertnaut
    Oct 23, 2022 at 23:36
  • $\begingroup$ @desertnaut No. $\endgroup$
    – schmixi
    Oct 24, 2022 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.