Improving precision and recall for imbalanced large data set

Question

I have a data set of 1 million points and 30 features. The output variable has multiple classes (1 to $n$) but the problem I'm interested in is only concerned whether the output belongs to class 1 or otherwise. I therefore converted the problem into a binary classification problem. Due to the large amount of data, using sklearn's histogram-based gradient boosting classifier offered a model that was fast to train especially for hyperparameter tuning using cross-validation.

But I realized that my data set is imbalanced (10% 1's and 90% 0's). Hence, I tried other classifiers to improve the performance. HistGradientBoostingClassifier currently does not have the balance weights option.

I tried a simple logistic regression with balanced class weights. I tried various regularization parameters but the precision was around 15% and recall around 62%. It did not change much as I varied the parameter making me think that a logistic regression model is insufficient.

I tried imbalanced learn's balanced bagging classifier on top of the histogram-based gradient boosting classifier and was able to get the precision to 20% and recall to 72% without any hyperparameter tuning. However, I cannot tune the hyperparameters of the histogram-based gradient boosting classifier, only those of the balanced bagging classifier. Doing a randomized search cross validation takes long but reasonable compared to using the histogram-based gradient boosting classifier only.

I am also trying RandomForestClassifier with class_weight='balanced_subsample'. Using n_estimators=100 and max_depth=10, I was able to obtain a precision of 25% and recall of 45%. The problem with this approach is that this set of parameters alone took 4 minutes, much longer than any of the methods above. It is therefore hard to do hyperparameter tuning with RandomizedSearchCV as it would take a long time.

Right now, I am almost running out of ideas as to what to try next to obtain a high precision and recall score. I am tempted to try SVMs since they should be relatively faster to train and are nonlinear in the original feature space, hopefully more expressive than logistic regression. I am also going to try imbalanced learn's balanced random forest classifier in case the model can be trained reasonably fast.

Does anyone have other suggestions for my problem? I have 2 issues mainly: 1) speed of training so I can tune the hyperparameters and 2) increasing precision and recall for this imbalanced data set.

Answer 1

Speedup Hyperaparameter Tuning :

1. Use Randomly Sample Data Sometimes if you have enough data to speedup hyperparameter tuning you can search best hyperparams on sampled data and use the best hyperpaarms on complete data.

2. Replace Grid Search with Smart Search Grid Search searches all the possible permutations and can take a lot of time. I would suggest use a Smart Search algorithm which search a few hyperaparam evaluate their quality and take next steps accordingly. It reduces the number of iterations and in most cases gives as good as results as Grid Search

3. Learning from best models params For example max depth parameter leads to exponential increase in time as 2**n decision needs to be made. So we very cautious on which parameters to chose and which range

Other methods to tackle imbalance are as follows :

1. Upsampling / Downsampling

2. Generate Synthetic Data Using Smote

3. Use Class Weights in RF

4. If you see some sample misclassified i will also suggest to explore sample weights in algorithms