I am working on an insurance use case to build a logistic regression classifier to predict if a policy will lapse or not. The dataset has more than 20 categorical features for a policy. Each categorical feature in itself can have multiple values out of which only one will be applicable to a policy at a time. e.g., there could be multiple ways of premium payment (like annually, half-yearly, quarterly or monthly). The policy will have either of these value attached to it. As part of feature reduction, I used chi-square test against my target feature (if a policy will lapse or not) and removed a few features. After this, I created dummy variables (0 or 1) for all of them and tried to find a correlation using .corr() function in pandas. This operation helped to identify some dummy features have a high correlation with target feature e.g a specific value of product type, renewal type, source of business. Using these dummy features variable I trained my logistic regression classifier which is giving an accuracy of 94%, precision of 81%, recall of 98% and area under AUC value of 0.98, which I believe is a case of model overfitting.
I believe, if I use this model on a policy which does not have categorical feature values on which it is trained, it will fail miserably.
Please guide me if it is correct to use individual values of categorical feature to train the model? And if this value is not present/applicable to a new data then the model will not be able to predict with a high level of accuracy. Please guide how is feature selection done in such a case and how should the model be trained. Will appreciate if link to similar implementations could be provided.