1
$\begingroup$

I have a dataset with 4.7k records and 60 features. 1558 records of indication label 1 and 3554 records indicating label 0. Am solving a binary classification problem. I used SMOTE on training data to balance the data

I applied various feature selection algorithms

1) RFECV - returns optimal 30-40 features based on model that I use

2) SelectFromModel - returns top n features based on what I keyed in. I tried n=15 and n=10

3) Kbest - returns top n features based on what I keyed in. I tried n=15 and n=10

My questions

1) The output from Feature selection algorithm 1,2,3 provides feature which are little bi different. I mean from algo 2 and algo 3, around 10 features are same but remaining 5 are different. What does this imply? Is it normal to see such differences? Or am I making any mistakes here?

2) I tried resampling the minority class using SMOTE and applied Gridsearchcv but still I don't see a good recall value for minority class(which is oversampled now). Please find the screenshot below for metrics

3) Does Gradientboosting, xtreamegradientboosting,catboost algorithms doesn't require us to select features? Are they intelligent enough to pick features and help us increase the accuracy? Should I pass the data (all features) as is

Can you help me as to how can I increase the metric values above 90 for both classes?

Currently am trying only with 15-20 features because certain features from feature selection output were correlated, so I removed them and using only 15-20 features.

Can you offer some suggestions please? CATboost is running for a long time.

enter image description here

$\endgroup$
2
$\begingroup$

Here are my two cents.

1) While there are some things that you may want to do differently (see part 2), what you observe is normal. If you had a trivial problem with a simple landscape, all of the methods would converge on exactly one subset of features. Feature selection in the real world, however, is rarely a trivial problem (as long as you are looking for "true" optimality and not a "good enough" solution). You are facing a search space of size $2^{60}$, which is impossible to assess in its entirety, so navigation methods will vary. RFECV is a top-down approach that attempts to shed as many weak features as possible based on the CV scores, SelectFromModel relies on estimator's internal objectives to do the selection, while SelectKBest in its default form is a simple filter that computes ANOVA F-values without considering feature interactions. Since the criteria are so different, you are bound to end up with different feature subsets -- each method operates on a landscape that is "distorted" by its objective function. The fact that the subsets you have found intersect suggests that the common features possess some sort of optimality (when all used approaches are considered). While you are at it, consider sequential feature selection methods, especially the floating approaches (both bottom-up and top-down), as they can be a great deal more effective than plain RFE / KBest.

2) You are using SMOTE to balance the training data. From your problem statement it seems that you are introducing synthetic data points before feature selection. If this is the case, then please consider the implications: for RFECV (if you do something like rfecv.fit(X_train, y_train)), the validation sample is not drawn from the original distribution, but from the one already affected by synthetic data, so the CV scores that drive the selection are not based on the true qualities of the features. This discrepancy may be too small to actually affect anything, but the fact remains. In your case, the class balance is 70/30, which is not that bad. Do you actually observe any increase in performance after balancing? As far as metrics go, the area under Precision-Recall curve is stable irrespective of the balance.

3) Decision trees (extensively used in many flavors, including XGBoost and CatBoost that you mentioned) have feature selection embedded into the process of learning. This means you can just plug in full featured data into the learner and have it figure out the most important features completely on its own. It's very effective and usually sufficient. However, in order to have variety of opinions within the ensemble, forest methods (most of them, anyway) will give all features an equal chance of being used, so it's very likely that you will end up with a situation where all features have some importance -- you will still have to decide which ones to keep. Weak features with noise that correlates with your labels by pure chance can affect your model's ability to generalize and they can even trick you into keeping them as useful. Also, when your sample size is limited (and it always is), increasing the number of features does not help the confidence of your assessments, either. Without enough observations, how can you be sure that the label is what it is because of this one feature and not ten others? So delegating it all to the forest without any preliminary or followup steps is not necessarily the best course of action.


The answer to the general question of the post: How to Maximize recall for Minority class?

This is quite straightforward. Define recall on minority class as your objective, i.e. let it be the output of the scoring function that you supply to your feature selection algorithm. For sklearn's feature selection methods it's the score_func parameter, for mlextend's SFS it's scoring. The resulting feature subsets are likely to contain something of interest.

Only after performing a thorough search for optimal features, selecting the best kind of model for your problem and having tuned its hyperparameters, if you still for whatever reason want to push your precision/recall further, you may want to look into stacking -- this will squeeze out the last bits of performance juice at the expense of model complexity. But the chances are that it's really not worth it.

$\endgroup$
  • $\begingroup$ I have a query with respect to second point. Can I make sure I understand correctly? Due to English proficiency issues. Currently what I did was 1) I split the data into train and test. 2) apply SMOTE only on training data 3) fit the model on resampled train data 4) validate it on original (not sampled) test data. In this case, validation sample is from the original distribution right? $\endgroup$ – The Great Dec 16 '19 at 1:14
  • $\begingroup$ In addition, are you saying that my precision recall scores look stable and isnt that bad? . $\endgroup$ – The Great Dec 16 '19 at 1:20
  • $\begingroup$ upvoted for the help and response. Would you mind to explian point 2 in much simpler terms. Apologies that I am new to ML and your inputs would definitely be helpful. Much appreciated $\endgroup$ – The Great Dec 16 '19 at 2:50
  • $\begingroup$ Can you let me know as to how would you do point 2? $\endgroup$ – The Great Dec 16 '19 at 2:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.