0
$\begingroup$

I am working on classification problem where the dataset contains 90% of features as categorical. It is binary classification problem, and the class is heavily imbalanced. I performed Smote over sample and created a model. I also tried similar approach with undersampling. Both the method with logistic regression performs mediocre. I want to know how having too many categorical variables impacts the model and possible efficient way to approach the problem

feature1:1-3
feature2:0-1
feature3:0-3
feature4: 1-4
feature5: 0-2
feature6: 0-5
feature7: 1-4
feature8: continuous( max 10)
feature9 continuous( max 10)
class: 0-1
Improve this question
$\endgroup$

3 Answers 3

Reset to default
2
$\begingroup$

You need to distinguish a few problems here. How well can your data classify some outcome, what features should be used, how to deal with imbalanced classes.

Categorical features are not a problem per se. In the current stage, feature selection might be an issue. Thus, use logit with lasso or ridge to shrink features which are not too helpful (happens automatically). Also dummy/one-hot encoding would be worth a try (jointly with lasso).

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeClassifier.html

Improve this answer
$\endgroup$
2
$\begingroup$

As already specified by @Peter, categorical features are not a problem per se provided that you encode them in the "right way" (problem specific). I see that you are already starting with integer encoding, but you are not satisfied with the results. Given your small number of categorical features and levels of each feature as a first step I'd try with one-hot encoding. Also notice that with integer encoding you are giving a natural order to your levels, e.g. feature3 = 3 is greater than feature3 = 1, so be careful about it.

Other than categorical features encoding and feature selection, your poor performance might also be caused by over and undersampling techniques that have not been tuned properly to preserve the original data distribution. I suggest to try weighting of the loss function as well to counteract imbalance.

Improve this answer
$\endgroup$
1
$\begingroup$

You can try to use CatBoost (https://catboost.ai). This library is developed specially for categorical features support and is based on gradient boosting on decision trees. Hope, it will help

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ you can work with categorical features in any method, so where is the point? $\endgroup$
    – Peter
    Commented Aug 9, 2019 at 7:43
  • $\begingroup$ @Peter you have just said that shrink features which are not too helpful and feature encoding are the right approach. And at the same time you asked, where is the point to use the library, which has already taken into account all this issues and implemented it on the algorithmic level? CatBoost combines multiple categorical features, provides split based on informativeness and gives rather powerful vector representation of categorical data. $\endgroup$
    – Lana
    Commented Aug 9, 2019 at 10:08
  • $\begingroup$ My question: why boosting? It could be a next step, but the question is framed under logit which is less expensive in terms of computation. $\endgroup$
    – Peter
    Commented Aug 9, 2019 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.