I see from reading that the medical industry is using machine learning to apply to small data set and wanted to understand how this worked? As such I created a small data set for my wife's business.

I have a data set that will be around 100k units, how to I determine the best predictive or factor model to use when approaching this small'ish data set? I want to avoid overfitting etc.

Any common pitfalls that I should avoid and/or tricks I should use to get more out of this data?


In general, Machine Learning algorithms handle volumes data. This doesn't mean that you cannot extract information from "small" data.

Keep in mind:

  • Overfitting. With only a few data, the risk to overfit your model is far higher.
  • Outliers. Those become nastier. If you have millions of data, a couple of outliers will not be a problem. But with only a few, they will definitely skew your results.

Models to use

To avoid overfitting, you need to avoid complexity. As a result, you need a model that has as less of parameters as you can. For example, linear/logistic regression could be what you need. Naive Bayes and SVM might work well, but this is not a generalization.

Cleaning up your data

Together with the outliers, you need to be careful with the rest of the cleaning process. Noise can skew a lot your data.

This is a nice blog post about "small data" and what you can do when you are working with them. Highly recommended.

What to do with “small” data? (7 min read)

About Outliers

I recommend the answer from this Data Science Stack Exchange question. Short story. You need to examine the outliers one by one. There isn't any rule written in stone. But you need to identify if those outliers are just wrong input data or data that makes sense to be there, even as an outlier. For example, if you have the sales of an eshop per day and in your data you have a specific day where the sales are 10x more, you need to understand if that was an error or something happened that day to increase the sales (maybe a discount campaign or email advertisement).

Also, it depends on what kind of problem you have. If your problem is an anomaly detection, then you don't want to remove the outliers.

Another suggestion (I don't have any source to link) is to update the values of those outliers with the mean, mode or another metric from your data. If you have only a few rows of them, removing a dozen will make it more difficult.

In general, this should be part of your model process. You try once with outliers, a few times without or with updated values and try to understand if it is underperformed or overfitted.

  • $\begingroup$ With regards to the outliers, do you suggest that they should be excluded completley from the data set and if so is there a standard process to do this? i.e. where do you draw the execulsion paramaters. Is there a specific 'noise' issue which I should be looking out for? $\endgroup$ – lonvangen Dec 30 '17 at 21:37
  • $\begingroup$ Please check my updated answer about outliers $\endgroup$ – Tasos Dec 30 '17 at 21:49
  • $\begingroup$ Thanks- The model I was thinking of using was the wapper model. Which I know is prone to overfitting so this is helpful. $\endgroup$ – lonvangen Dec 30 '17 at 21:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.