I am working on classification problem where the dataset contains 90% of features as categorical. It is binary classification problem, and the class is heavily imbalanced. I performed Smote over sample and created a model. I also tried similar approach with undersampling. Both the method with logistic regression performs mediocre. I want to know how having too many categorical variables impacts the model and possible efficient way to approach the problem

feature4: 1-4
feature5: 0-2
feature6: 0-5
feature7: 1-4
feature8: continuous( max 10)
feature9 continuous( max 10)
class: 0-1

You need to distinguish a few problems here. How well can your data classify some outcome, what features should be used, how to deal with imbalanced classes.

Categorical features are not a problem per se. In the current stage, feature selection might be an issue. Thus, use logit with lasso or ridge to shrink features which are not too helpful (happens automatically). Also dummy/one-hot encoding would be worth a try (jointly with lasso).




As already specified by @Peter, categorical features are not a problem per se provided that you encode them in the "right way" (problem specific). I see that you are already starting with integer encoding, but you are not satisfied with the results. Given your small number of categorical features and levels of each feature as a first step I'd try with one-hot encoding. Also notice that with integer encoding you are giving a natural order to your levels, e.g. feature3 = 3 is greater than feature3 = 1, so be careful about it.

Other than categorical features encoding and feature selection, your poor performance might also be caused by over and undersampling techniques that have not been tuned properly to preserve the original data distribution. I suggest to try weighting of the loss function as well to counteract imbalance.


You can try to use CatBoost (https://catboost.ai). This library is developed specially for categorical features support and is based on gradient boosting on decision trees. Hope, it will help

  • 1
    $\begingroup$ you can work with categorical features in any method, so where is the point? $\endgroup$
    – Peter
    Aug 9 '19 at 7:43
  • $\begingroup$ @Peter you have just said that shrink features which are not too helpful and feature encoding are the right approach. And at the same time you asked, where is the point to use the library, which has already taken into account all this issues and implemented it on the algorithmic level? CatBoost combines multiple categorical features, provides split based on informativeness and gives rather powerful vector representation of categorical data. $\endgroup$
    – Lana
    Aug 9 '19 at 10:08
  • $\begingroup$ My question: why boosting? It could be a next step, but the question is framed under logit which is less expensive in terms of computation. $\endgroup$
    – Peter
    Aug 9 '19 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.