0
$\begingroup$

Does anyone know any existing research or have observed some experimental results in deep learning about the following:

For a fixed data set, if you subset the dataset from 50% to 90% as training set, using the same number of epochs, will the training error be bigger as the training set becomes bigger?

would be great if anyone could give some concrete experimental results or paper about this? For example, this one https://arxiv.org/abs/1901.04169v1 and this one https://www.researchgate.net/publication/333671691_Resampling-based_Assessment_of_Robustness_to_Distribution_Shift_for_Deep_Neural_Networks

but both papers have not drawn any conclusions on this issue.

$\endgroup$
3
$\begingroup$

I don't think there a connection between size of the training set and model's performance during training. A dataset can be very noisy and unpredictable, independently from its size.

To my knowldge, it is not possible to derive any general rule about this.

$\endgroup$
  • $\begingroup$ any papers or have you observed any phenomena about this? $\endgroup$ – sunxd Jul 18 at 10:46
3
$\begingroup$

This question cannot generally be answered, because the training-accuracy is a result of the models' ability to generalize and its ability to overfit.

For example: On one hand, if you decrease the size of your dataset, your training-accuracy might increase, since the model could start learning the random noise of your data (aka. overfit). On the other hand, your training-accuracy could also decrease since the model gets fewer data to learn from and might not generalize as well.

$\endgroup$
  • $\begingroup$ any papers or have you observed any phenomena about this? $\endgroup$ – sunxd Jul 18 at 10:46
  • $\begingroup$ @sunxd This is just logic reasoning, experience, and basics from the university. Related papers should be from the '70s and '80s. $\endgroup$ – georg-un Jul 18 at 10:53
  • $\begingroup$ thanks, any keywords to search for these papers from 70s and 80s? $\endgroup$ – sunxd Jul 18 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.