Machine Learning Binary Classification Model on a Small Tabular Imbalanced Dataset - Improving Performance

Question

I have a dataset that is fairly small (15,000 rows), with 10 features for a model to learn from. It is not possible to increase the size of this dataset. I am using machine learning for binary classification on this dataset; it is severely imbalanced, with around 2% for the positive class and 98% for the negative class. As stated in the title, this is the basic tabular dataset (with numerical/continuous features, categorical features, and binary features); there are around 30% missing values for one column and 2% missing values for the other. I am trying to improve the performance of the machine learning models that I have tried, as they have done fairly poor (please see below).

In terms of preprocessing, I am doing one-hot encoding for the categorical features and StandardScaler for the numerical features. I am imputing the missing values for the 2% NA column with multiple imputation; for the 30% NA column, I am just imputing with a string "none". To deal with the class imbalance (as that may be a problem since the data is extremely imbalanced and is small; please see Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?), I am trying various resampling techniques: random undersampling, random oversampling, SMOTE, SMOTE-ENN, and SMOTE-Tomek Links. To avoid any data leaks, everything is in a pipeline.

I start off with K-Fold cross validation (specifically 10 folds) as a baseline. Then, I do RandomizedSearchCV (with a good amount of iterations) to tune the hyperparameters of the models. The models that I have tried Random Forest, XGBoost, LightGBM, SVM (with RBF and linear kernels), CatBoost, and TabNet. The performance of these models basically are:

• RandomizedSearchCV: In terms of accuracy, there is no overfitting (around 1% difference between train and test). However, as this is an imbalanced dataset and as accuracy is an improper scoring rule, the accuracy doesn't really matter (it is around 60%-70% for the various models). As this is a medical dataset, recall seems to matter the most, so hyperparameter tuning was done to optimize recall. Recall is around 70% for the various models. Somewhat because we are optimizing recall, F1 and precision are very low, around 40% and 50%, respectively. For the ROC-AUC score, it is the high 70% to low 70% for the various models, which seems to be better, but still not very good. The area under the Precision-Recall Curve (AUPRC) score is terrible; it is around 0.05, where a model by just chance would be 0.02 in my case.
• K-Fold Cross Validation as a baseline: both train and test accuracy are in the 70s (still doesn't really matter, though). Recall is sometimes around 60-70% or around 40-50% depending on the specific data resampling technique (I can see that the hyperparameter tuning does improve recall). AUC is fair, but not good, also.

Basically, my questions are:

1. What are some techniques that I should try to improve the performance of my model? Should I try dimensionality reduction (e.g. PCA)? Should I remove the column with the 30% NA values from the dataset? Should I not resample my dataset? Should I try a different imputation technique (e.g. MiceForest)?
2. Are there any models I should try to improve performance? Should I try more complex neural networks (e.g. convolutional NN's, deep fully-connected NN's, fully-connected NN's with residual blocks, LSTM's...)?
3. Should I do hyperparameter tuning to optimize recall? It is an improper scoring rule; should I instead do hyperparameter tuning to optimize AUC, for example, which is a semi-proper scoring rule, or should I optimize the Brier score (once I implement it), which is a proper scoring rule?
4. Instead of doing 10-Fold cross validation, should I do 5-Fold cross validation since this is a small dataset?

Also, training/preprocessing time doesn't really matter to me.

Thank you very much for any suggestion or help!

Answer 1

I don't think PCA will be particularly helpful in your case since you only have 10 features, unless your one-hot encoding is creating hundreds of additional features or the 10 features themselves are highly collinear. Regarding your imbalanced dataset, it's definitely necessary to take measures to make it more balanced. Techniques like class weight balancing, under/over-sampling, are worth trying. In my experience, SMOTE hasn't had much success outside of academia, but that might just be specific to my industry.

Feel free to experiment with neural network models, but don't expect drastically better results compared to the tree-based models you've already used. You mentioned LSTM, but it's really only useful if your dataset is a time series or has some chronological structure.

In medical datasets, recall is probably the most important metric to optimize for, but nothing is stopping you from optimizing for both recall and AUC at the same time. Just be cautious about which dataset you're using for hyperparameter tuning—if you're using your test set, it might lead to indirect data leakage.

Regarding cross-validation, 10-fold is probably fine, but it shouldn't matter too much since a model that performs well on 5-fold cross-validation should also do well on 10-fold, though there are exceptions.

Additional things you can do include error analysis to see if there are any systemic issues, and performing data analysis/error analysis. For example, calculate AUCs per class, check if there are contradictory rows in the dataset (i.e., rows with similar features but different target values), and see if these differences make sense. If they don't, figure out why, as they could be outliers affecting your model.

Also, examine how well each feature correlates with the target, look at the most important features from Random Forests, and manually verify if they make sense. Check if you've overfitted the dataset, and look at the standard deviation of your 10 folds to see if your model is stable across folds. Addressing these questions and following good data science practices should lead to a better model.

However, remember that simply throwing different models at the data and seeing which one sticks can only get you so far. To build a truly good model, you need to understand your data deeply and address any underlying issues. The quality of your model will ultimately be limited by the explainability in your data—if your data lacks certain information, no model can learn it. Additionally, consider looking at winning solutions from Kaggle competitions for similar problems to gain further insights.