ML Project - Achieve 2 Objectives

I have a dataset with 5K records focused on binary classification. I am posting it here to seek your suggestions on project methodology

Currently what is my objective is

1) Run statsmodel logistic regression to find risk factors that influence the outcome

2) Then build a predictive model based on best features (may or may not include risk factors). because as you may know not all significant variables are good predictors.

Though I can use scikit-learn logistic regression to build a predictive model but I am planning to go with Xgboost because it provides better performance in my dataset (non-linear data slightly imbalanced)

I do step one because I have to find what are the risk factors that influence the outcome, so I am doing it. (ex: risk factors that influence customer to default in loan repayment) You know where we get p-values and find significant risk factors.

In 2nd step, I build predictive model because I realized through running the built model that not all risk factors are good predictors. So in the end, I include new set of features that help in better prediction along with risk factors

Do you think that I am right in having/approaching this as two objectives problem?

Do you think what I am doing is redundant or am proceeding in the right direction?

Do you think there is no reason to use 2 algorithms separately?

Do you have any suggestions or tips to make it easy to achieve my objective?

1 Answer

Xgboost does the feature selection for you. If you want to report how valuable certain features are for prediction, print the feature importances. Those will, however, just tell you "feature $$x_1$$is very important for predicting the outcome, feature $$x_2$$ is nearly useless for predicting the outcome, etc.". To get risk factors with p-values, you need a more interpretable model. You have to use linear classification, for example. Then you can make statements like "high $$x_1$$ correlates with a positive outcome".

If you really want to only use a subset of features, train an Xgboost model on a validation dataset and drop features that have low importance. Then run an Xgboost model with the remaining features on the remaining training set.

You need to think about what you want: Do you want to explain observations and extract explicit knowledge from data and try to find possible causal links? Use linear methods. Do you want to predict an outcome for new patient data? Use gradient boosting. You can obviously do both.

• Thanks. Upvoted.. Bit for risk factors, we usually look at p values right? Jan 5 '20 at 23:51
• Then in this case, don't you think it's essential to break this into two separate objectives. 1) for risk factors that can explain the outcome 2) as not all risk factors are good predictors, don't I have to try to find best predictors... Because in medicine literature, don't they usually look for some explanation when we do and so are the risk factors. I assume p values can help to convince regarding the risk factors. And for better prediction ability, I could use Xgboost instead of Scikit learn logistic regression because the former gives better performance metrics.can help on this plz? Jan 6 '20 at 1:37
• You are right, I was not aware of the definition of risk factors. Gradient boosting models like XGboost will not provide you with p-values nor tell you how the features influence the outcome. You need to think about what you want: Do you want to explain observations and extract explicit knowledge from data and try to find possible causal links? Use linear methods. Do you want to predict an outcome for new patients' data? Use gradient boosting. Jan 6 '20 at 9:38
• Most ML techniques deal with correlation not causation. The question of identifying risk factors, or causal variables, is entirely different than what is usually done in ML. Jan 6 '20 at 9:44
• @Icrmorin That is why I said "possible causal links" and not "causal links". Obviously experiments are needed. People found that smoking correlates with lung cancer and then started to find out if there is a causal link using scientific experiments. Jan 6 '20 at 9:48