I have a binary classification problem with 5K records and 60+ features/columns/variables. dataset is slighlt imbalanced with 33:67 class proportion

What I did was

1st) Run a logistic regression (statsmodel) with all 60+ columns as input (meaning controlling confounders) and find out the significant risk factors (p<0.0.5) from result(summary output). Because I have to know that my risk factors are significant as well i.e causal modeling. Meaning build a predictive model on the basis of causal features. I say this because in a field like medical science/clinical studies, I believe it is also important to know the causal effect. I mean if you wish to publish in a journal, do you think we can just list the variables based on feature importance approach (results of which differ for each FS approach). Ofcourse, I find some common features across all feature selection algorithm. But is this enough to justify that this a meaningful predictor? Hence, I was hoping that p-value would convince and help people understand that this is significant predictor

2nd) Use the identified 7 significant risk factors to build a classification ML model

3rd) It yielded an AUC of around 82% in test data (30% of split from full dataset)

Now my question is

1) Out of 7 significant factors identified, we already know 5 risk factors based on domain experience and literature. So we are considering the rest 2 as new factors which we found. Might be because we had a very good data collection strategy (meaning we collected data for new variables as well which previous literature didn't have)

2) But when I build a model with already known 5 features, it produces an AUC of 82.1. When I include all the 7 significant features, it still produces an AUC of 82.1-82.3. Not much improvement. Why is this happening?

3) If it's of no use, how does statsmodel logistic regression identified them as significant feature (with p<0.05)?

4) I guess we can look at any metric. As my data is slightly imbalanced (33:67 is the class proportion), I am using only metrics like AUC and F1 score.

5) Should I balance the dataset because I am using statsmodel Logistic regression to identify the risk factors from the summary output? Because I use tree based models later to do the classification which can handle imbalance well, so I didn't balance.Basically what I am trying to know is even for `significant factor identification using statsmodel logistic regression, should I balance the dataset?

6) Can you let me know what is the problem here and how can I address this?

7) How much of an improvement in performance is considered as valid/meaningful to be considered as new findings?


You might wonder why do I have to do Xgboost when I do logistic regression in the first place. I do this because with Logistic regression yields only an AUC of 79-80 whereas Xgboost results with an AUC of 81-82. Reason why I do Logistic regression in the first place is because I need to find a way to justify that these variables really explain the outcome. Hence thought choosing p-value might be best approach which is usually found using Logistic regression. Problem with Feature selection approach using different algorithms is it returns different subsets with different ranking order of features. Though I find common features across all FEATURE SELECTION algorithms, I feel there is no way to justify that these variables really explain the outcome. They might have good predictive power but do are they also responsible for causing a change in the outcome? Hope I made my objective clear. Request you to let me know for any questions

  • $\begingroup$ Your data isn't imbalanced imo, try giving higher class_weight to the class you care about. $\endgroup$
    – Aditya
    Jan 1, 2020 at 1:35
  • $\begingroup$ As I am using 'Xgboost', I make sure to give the parameter 'scale_pos_weight' appropriately based on the ratio of positive and negative class.. Is there any thing else that I can do? $\endgroup$
    – The Great
    Jan 1, 2020 at 4:50
  • $\begingroup$ Are you saying that I should look at accuracy metric as it isn't that bad when it comes to class balance? $\endgroup$
    – The Great
    Jan 1, 2020 at 4:52
  • $\begingroup$ Does your domain imply different cost for misclassifying an example with regards to the two classes (i.e. a false negative being worse than a false positive or vice versa)? $\endgroup$
    – Jonathan
    Jan 1, 2020 at 15:28
  • $\begingroup$ Yes, we could say that. It's always costly to let a person with disease as disease free.. Meaning false negative is costly and we should minimize it.. $\endgroup$
    – The Great
    Jan 1, 2020 at 20:57


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.