3
$\begingroup$

I am trying to collect some data for ML, specifically for training a neural network model, and I don't know how big the data set is enough.

So is there a rule of thumb on how many data of dimension DIM should one collect for training a NN-Model ? For example, it depends on the number of features or the kind of NN-Model or something else ?

Any help will be appreciated.

$\endgroup$
3
$\begingroup$

In this video by Caltech prof. Yaser Abu-Mostafa, he explains the relationship between dimension of a dataset and it's size required for any learning model to work.

As a general rule of thumb, size of dataset should be at-least about 10x it's dimension and should be independent of the model used.

Also, this link has summaries from some of the relevant papers, viz.

For a finite sized data with little or no a priori information, the ratio of the sample size to dimensionality must be as large as possible to suppress optimistically biased evaluations of the performance of the classifier.

This says, the ratio of size of dataset (sample) to dimension should be as large as possible to reduce classifier bias towards a particular class.

The ratio of the sample size to dimensionality should vary inversely proportional to the amount of available knowledge about the class conditional densities.

In a classifier setting, the more knowledge we have for each class' probability density, lesser can be the sample-size to dimension ratio.

In simpler terms, we can say we should be including as much data as possible and if we are not able to do that, include as much information as possible in the small dataset itself, because for any model to work we need to feed it with high variance dataset.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.