I have a binary classification problem with 5K records and 60+ features/columns/variables. dataset is slighlt imbalanced with 33:67 class proportion
What I did was
1st) Run a logistic regression (statsmodel) with all 60+ columns as input (meaning controlling confounders) and find out the significant risk factors (p<0.0.5) from result(summary output). Because I have to know that my risk factors are significant as well i.e causal modeling. Meaning build a predictive model on the basis of causal features. I say this because in a field like medical science/clinical studies, I believe it is also important to know the causal effect. I mean if you wish to publish in a journal, do you think we can just list the variables based on feature importance approach (results of which differ for each FS approach). Ofcourse, I find some common features across all feature selection algorithm. But is this enough to justify that this a meaningful predictor? Hence, I was hoping that p-value would convince and help people understand that this is significant predictor
2nd) Use the identified 7 significant risk factors to build a classification ML model
3rd) It yielded an AUC of around 82% in test data (30% of split from full dataset)
Now my question is
1) Out of 7 significant factors identified, we already know 5 risk factors based on domain experience and literature. So we are considering the rest 2 as new factors which we found. Might be because we had a very good data collection strategy (meaning we collected data for new variables as well which previous literature didn't have)
2) But when I build a model with already known 5 features, it produces an AUC of 82.1. When I include all the 7 significant features, it still produces an AUC of 82.1-82.3. Not much improvement. Why is this happening?
3) If it's of no use, how does statsmodel logistic regression identified them as significant feature (with p<0.05)?
4) I guess we can look at any metric. As my data is slightly imbalanced (33:67 is the class proportion), I am using only metrics like AUC and F1 score.
5) Should I balance the dataset because I am using statsmodel Logistic regression to identify the risk factors from the summary output? Because I use tree based models later to do the classification which can handle imbalance well, so I didn't balance.Basically what I am trying to know is even for `significant factor identification using statsmodel logistic regression, should I balance the dataset?
6) Can you let me know what is the problem here and how can I address this?
7) How much of an improvement in performance is considered as valid/meaningful to be considered as new findings?
POST UPDATE
You might wonder why do I have to do Xgboost when I do logistic regression in the first place. I do this because with Logistic regression yields only an AUC of 79-80 whereas Xgboost results with an AUC of 81-82. Reason why I do Logistic regression in the first place is because I need to find a way to justify that these variables really explain the outcome. Hence thought choosing p-value might be best approach which is usually found using Logistic regression. Problem with Feature selection approach using different algorithms is it returns different subsets with different ranking order of features. Though I find common features across all FEATURE SELECTION algorithms, I feel there is no way to justify that these variables really explain the outcome. They might have good predictive power but do are they also responsible for causing a change in the outcome? Hope I made my objective clear. Request you to let me know for any questions
