How big should training data be?

Question

I am solving a problem connected with medicine and from each patient I get about 100 features. It's a classification diseases problem, however measurements take a lot of time and also require money.

What is the optimal size of training data? I understand that I should collect as much data as possible, however is there any empirical advice?

Answer 1

While the other answers aren't wrong, they don't touch anything about bioinformatics. I'll go into details.

In bioinformatic, simply asking for the size of the training set makes no sense. You'll need to understand deeper about bioinformatic to answer this question. There is a good reference: An ensemble approach to accurately detect somatic mutations using SomaticSeq for you.

The goal of the project is to build a gradient-boosting machine for classifying somatic mutation given an alignment file. This is very close to what you're doing.

Now, you want to know the training size? You should ask yourself the following:

• Sequencing errors
• Reads
• Sequencing depth
• Sensitivity
• F1 score
• Allele frequency

In an next-generation sequencing, your sequencing machine always give you errors. How reliable is your pipeline to detect those errors?

How good are your reads for your problem? For example, if you're detecting disease related to structural variant, you might need more samples to compsenate for the short reads.

What's your sequencing depth? In the paper I quote, they have like 30x depth. It'd be pointless to talk about the training size unless you know well about your depth.

Does your sequencing cover well your region of the genome you're interested in?

Sensitivity and F1 score. You can have good idea by drawing an F1 curve something like as in the paper:

What's your expected allele frequency range? If you have something like close to 0.5 (ie: maximum heterozygosity), you'll probably be safe with a smaller sample size. However, if you're trying to detect a rare disease, something like <0.1 as in the diagram, you'll need to have more samples and most likely more features.

Empirically, we generally use spike-in standards to measure the sensitivity. Once you draw the sensitivity against training size, you'll know at what training size to use achieve whatever sensitivity you're looking for. A typical tool would be ROC.

Answer 2

The ratio between no of samples and features should be more than 10:1 to get sufficiently decent results. But this ratio also varies with applications. If training data is very less : make sure that dataset is balanced, try ensemble methods, careful feature selection, perform cross validation

Answer 3

Primarily, the concern with data-size arises because of the error that it can cause to the overlying model. With that concept in mind, you might want to measure the Bias and Variance of the model with your current data, understand the fitness of your model, and then take it from there.

Here's an article that I wrote that may help you do the same: Machine Learning: How Big Should Your Data Be?

This article talks about

1. Bias and Underfitting
2. Variance and Overfitting
3. Bias-Variance Trade-Off and Optimal Complexity

It concludes that:

• More sample data is always powerful, but is not always the right answer.
• As you increase your sample data, make sure to adjust your parameter space to minimize error.