2
$\begingroup$

I am solving a problem connected with medicine and from each patient I get about 100 features. It's a classification diseases problem, however measurements take a lot of time and also require money.

What is the optimal size of training data? I understand that I should collect as much data as possible, however is there any empirical advice?

$\endgroup$
  • $\begingroup$ what type of data are we looking at ? knowing this will affect greatly the answers you will get. Is it DNA analysis ? symtomatic measurements and observations ? Lab test results ? $\endgroup$ – Newtopian Apr 15 '16 at 17:04
  • $\begingroup$ There are some great answers, I just want to add something specific to medicine. Depending on the field, you can still publish results from ~20 patients, especially if the field is young (see texture analysis of medical images). $\endgroup$ – Hobbes Jul 14 '16 at 21:46
3
$\begingroup$

While the other answers aren't wrong, they don't touch anything about bioinformatics. I'll go into details.

In bioinformatic, simply asking for the size of the training set makes no sense. You'll need to understand deeper about bioinformatic to answer this question. There is a good reference: An ensemble approach to accurately detect somatic mutations using SomaticSeq for you.

The goal of the project is to build a gradient-boosting machine for classifying somatic mutation given an alignment file. This is very close to what you're doing.

Now, you want to know the training size? You should ask yourself the following:

  • Sequencing errors
  • Reads
  • Sequencing depth
  • Sensitivity
  • F1 score
  • Allele frequency

In an next-generation sequencing, your sequencing machine always give you errors. How reliable is your pipeline to detect those errors?

How good are your reads for your problem? For example, if you're detecting disease related to structural variant, you might need more samples to compsenate for the short reads.

What's your sequencing depth? In the paper I quote, they have like 30x depth. It'd be pointless to talk about the training size unless you know well about your depth.

Does your sequencing cover well your region of the genome you're interested in?

Sensitivity and F1 score. You can have good idea by drawing an F1 curve something like as in the paper:

enter image description here

What's your expected allele frequency range? If you have something like close to 0.5 (ie: maximum heterozygosity), you'll probably be safe with a smaller sample size. However, if you're trying to detect a rare disease, something like <0.1 as in the diagram, you'll need to have more samples and most likely more features.

Empirically, we generally use spike-in standards to measure the sensitivity. Once you draw the sensitivity against training size, you'll know at what training size to use achieve whatever sensitivity you're looking for. A typical tool would be ROC.

$\endgroup$
1
$\begingroup$

The ratio between no of samples and features should be more than 10:1 to get sufficiently decent results. But this ratio also varies with applications. If training data is very less : make sure that dataset is balanced, try ensemble methods, careful feature selection, perform cross validation

$\endgroup$
1
$\begingroup$

Primarily, the concern with data-size arises because of the error that it can cause to the overlying model. With that concept in mind, you might want to measure the Bias and Variance of the model with your current data, understand the fitness of your model, and then take it from there.

Here's an article that I wrote that may help you do the same: Machine Learning: How Big Should Your Data Be?

This article talks about

  1. Bias and Underfitting
  2. Variance and Overfitting
  3. Bias-Variance Trade-Off and Optimal Complexity

It concludes that:

  • More sample data is always powerful, but is not always the right answer.
  • As you increase your sample data, make sure to adjust your parameter space to minimize error.
$\endgroup$
  • 1
    $\begingroup$ That is a good article with really helpful visualizations. Any time patients are involved, data is often scarce and so the model/ feature selection becomes really important. $\endgroup$ – Hobbes Jul 14 '16 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.