Binary Classification with Very Small Dataset (<40 samples)

Question

I'm trying to perform binary classification on a very small dataset, consisting of 3 negative samples and 36 positive samples. I've been testing different models from scikit-learn (logistic regression, random forest, svc, mlp). Depending on random_state when using train_test_split, the train or test set might not have a negative sample in it and classification performance is poor because of this. I've read into oversampling techniques using ROSE or various flavors of SMOTE, but have also read that oversampling will lead to overfitting or does not increase performance. I had experimented with oversampling the training set and depending upon how the data is split into train/test the different models are each able to correctly classify unseen data (except for log reg). However, because of the possibility of overfitting due to oversampling I am unsure of the model's actual ability to perform on unseen data.

When not oversampling and just performing feature selection, tuning hyperparameters (e.g., class weights), and using LOOCV the models (not log reg) are able to correctly classify each sample as negative or positive. However, I have read that LOOCV tends to have high variance and I am unsure of how the classifiers would perform on new unseen data.

Unfortunately collecting more data is not possible, I have to work with what I currently have. My question is how do I approach the problem to achieve the best performance I can without overfitting the classification models? Having someone classified falsely as negative is preferable to having something classified falsely as positive. If the models are able to correctly classify everything when performing LOOCV is that the last step in the process before model deployment, or are there other things I should look into as well?

Answer 1

I'm not sure this will be a comprehensive answer but an opinion to give a push to the reasoning. There are only 3 negative cases. I could create a custom cross validation scheme: create a test case with one negative case and the rest 2 of them to put to the train set. Then iterate through the negative cases, giving a chance to everyone to be in the test set. Any test set I would enrich with positive cases keep the ratio of positive/negative cases fixed: 36/3 * 1 = 12 positive observations in each test set. I'm not sure this technic would work in any way but at least this can be a solution for the CV scheme problem.

I would definitely be prepared that the problem doesn't have any adequete solution with so poor data. I stress this idea in order to make a reasonable expectation on time, money budget as well as risks of the project.

I'm not sure it's reasonable to do so intensive overfitting. You may approximately count how many times you used any specific negative class observation in order to get a feeling of the degree of contamination of your model with overfitting. This is not strict or correct terminology, I just want to share my intuition. Each time you use the negative observation for training or assessment you increase you changes that a good model will fool you eventually. You have high risks with so few examples.

You may treat the problem as an anomaly detection problem. Split a observations in train-test set. For example, 10 observations in the test set, 3 of them are negative cases. Train a clustering model then look how it works on the test set. Does it group negative cases in one separate group or not? https://scikit-learn.org/stable/modules/clustering.html

Another approach is to add your own knowledge of the world to the problem if this is applicable to your case. For example, imagine we have titanic data, only 39 observations with 3 survivors and 2 columns: Name and Survived. I could suggest that gender is important and create a new column based on my world knowledge. Looks like I'm reinventing the wheel and feature engineering but anyway this may be useful for you.

The last point is that when you have so few data use data visualization intensively. Make your own, maybe even hardcoded if-else model, based on the plots where you painting data points by target (color=target). This could be more reliable and less prone to overfitting comparing with CV and complex models.

Answer 2

Maybe with data as small as 36 and only 3 positive samples the way is try to do a simple rule instead of a model, because it looks like a machine learning approach wouldn't work. But if you really want modelling one way is using Bayesian logistic regression because with this you can estimate the distribution of the betas and see the broadness of the intervals, or perform a simple logistic regression and bootstrap the sample in order to estimate the beta's intervals and any another metric of your interest.