4
$\begingroup$

I started to study and programming in neural networks for a little while now, but I never read about the minimum number of observations one must collect in a dataset to get robust results. Of course, more observations better results, but, Does exist an empirical or theoretical relationship between variables and observations number?

I mean, neither in econometrics you can compute the minimum number of observations, but it does exist some rule of thumbs that relies the number of exogenous variables to the target variable.

I wonder if there is something similar to that in neural networks too, but, till now, browsing on the internet, I did not find anything of useful.

Any ideas, advises or hint will be appreciated.

$\endgroup$
3
$\begingroup$

A neural network is nothing but a set of equations. And the basic rule of any set of equations is that you must have as many data points as the number of parameters.

The parameters of any neural network are its weights and biases.

So that means that as the neural network gets deeper and wider, the number of parameters increase a lot, and so must the data points.

This being said, the more proper and detailed way to know whether the model is overfitting is to check if the validation error is close to the training error. If yes, then the model is working fine. If no, then the model is most likely overfitting and that means that you need to reduce the size of your model or introduce regularization techniques.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Firstly, thanks for the answer and explanation (+1), @Azrael. So, according to you, shouldn't I care about the number of observation? Could you provide some reference about that? $\endgroup$ – Quantopik Jun 24 '15 at 21:05
  • 1
    $\begingroup$ The data points mean the observations. $\endgroup$ – Azrael Jun 24 '15 at 21:17
  • $\begingroup$ No problem. Happy to help. $\endgroup$ – Azrael Jun 24 '15 at 21:22
  • $\begingroup$ Maybe someone else knows better, but I"m not entirely sure this is true. You aren't trying to solve the equations of the neural network in a classical sense. You could quite easily have a complex network that was trained to recognise the difference between only two data points. $\endgroup$ – image_doctor Jun 24 '15 at 21:33
  • $\begingroup$ Yes, you are right. For example, you can keep giving a neural network 0 as an input, and 1 as an output multiple times and it will start learning to become a NOT gate. But what the question has demanded is a rule of thumb. The number of parameters that are required can not be correctly predicted without knowing more details. $\endgroup$ – Azrael Jun 24 '15 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.