Performing machine learning on small datasets

Question

As a beginner at machine learning, I wanted to work on a small project in which the dataset has only 80 rows and 5 columns. The dataset I am working with is related to a medical condition, with 4 columns as biomarkers, and the 5th column indicates whether the row (a patient) has the condition or not. So far, I have fitted the following 5 models (with accuracy and MCC scores):

• KNN (Accuracy: 43.5%, MCC:-0.164)
• Logistic Regression (Accuracy: 65.2%, MCC: .312)
• SVM (Accuracy: 60.9%, MCC: .214
• Random Forest (Accuracy: 86.95%, MCC: .769)
• Decision trees (Accuracy: 65.2%, MCC: .312)

I have used 5-fold cross validation to prevent overfitting, and yet most of my models are underperforming. I was also considering ensembling and bootstrapping, but with these lacking results, I am not sure how effective they would be. Do you have any tips concerning either:

1. Better algorithms for small datasets

2. Improvements I could make on the algorithms I have so far

3. Another method (e.g. Regularization)

Answer 1

With 5-fold cross validation, in each fold you're reducing your training dataset to 64 observations and evaluating against 16 observations. Assuming your data is balanced and you're stratifying folds, you're only giving your models 32 observations from each class to learn from, and misclassification of a single test set observation results in a 6.25% point change in accuracy for that fold. Even if it's classified correctly in the other folds, that single misclassification will still have a 1.25% point change on the across-folds average. Yikes. It should be no surprise your models aren't performing well: not only do they not have much data to train on, but your evaluation methodology is extremely unforgiving.

So yeah, your data is pretty darn small. Off the top of my head, I can think of a handful of general strategies you can potentially use to address this:

1. Use more of your data for training. Instead of 5-fold CV, try leave-one-out or bootstrap.
2. Generate fake data. This probably won't get you very far, but it's at least an option. Check out the SMOTE algorithm.
3. Transfer learning. Depending on what you're doing, it might be possible to leverage a pre-trained model and then tweak it slightly to suit your needs. If this is an option it can be extremely powerful, but chances are this won't be something you can reasonably pursue. Anyway, here's an article demonstrating this on another small data medical example: a pre-trained general purpose image classifier was trained to detect cancer from just 600 images! https://arxiv.org/abs/1711.10752
4. Go Bayesian. Bayesian methods allow you to incorporate outside information as a "prior belief". If you have subject matter expertise (or better yet, citations) that you can use to set your expectations for values of the model parameters or hyperparameters, bayesian methods will allow you to incorporate that information explicitly, and this can in turn help your model find good parameters faster since it doesn't need to learn everything about your problem directly from the little data available to it. If you're not careful, this can result in you giving yourself the model you want to have, in which case your evaluation metrics may not be as informative as you think they are. Here there be dragons: this approach is powerful, but it can be hard to do correctly, and playing with the prior is basically an invitation for people to be skeptical of your methods even if what you're doing is sound.
5. Get more data. I'm guessing this isn't an option or you would have done so already. But if there's a chance it's out there: go find it. Try poking around the literature associated with this condition, maybe you'll get lucky and find a public dataset. If you're feeling bold, you could try emailing other researchers and just ask them politely if you can use their data.