I'm trying to perform binary classification on a very small dataset, consisting of 3 negative samples and 36 positive samples. I've been testing different models from scikit-learn (logistic regression, random forest, svc, mlp). Depending on random_state when using train_test_split, the train or test set might not have a negative sample in it and classification performance is poor because of this. I've read into oversampling techniques using ROSE or various flavors of SMOTE, but have also read that oversampling will lead to overfitting or does not increase performance. I had experimented with oversampling the training set and depending upon how the data is split into train/test the different models are each able to correctly classify unseen data (except for log reg). However, because of the possibility of overfitting due to oversampling I am unsure of the model's actual ability to perform on unseen data.
When not oversampling and just performing feature selection, tuning hyperparameters (e.g., class weights), and using LOOCV the models (not log reg) are able to correctly classify each sample as negative or positive. However, I have read that LOOCV tends to have high variance and I am unsure of how the classifiers would perform on new unseen data.
Unfortunately collecting more data is not possible, I have to work with what I currently have. My question is how do I approach the problem to achieve the best performance I can without overfitting the classification models? Having someone classified falsely as negative is preferable to having something classified falsely as positive. If the models are able to correctly classify everything when performing LOOCV is that the last step in the process before model deployment, or are there other things I should look into as well?