does smaller training set always lead to better training accuracy?

Question

Does anyone know any existing research or have observed some experimental results in deep learning about the following:

For a fixed data set, if you subset the dataset from 50% to 90% as training set, using the same number of epochs, will the training error be bigger as the training set becomes bigger?

would be great if anyone could give some concrete experimental results or paper about this? For example, this one https://arxiv.org/abs/1901.04169v1 and this one https://www.researchgate.net/publication/333671691_Resampling-based_Assessment_of_Robustness_to_Distribution_Shift_for_Deep_Neural_Networks

but both papers have not drawn any conclusions on this issue.

Answer 1

I don't think there a connection between size of the training set and model's performance during training. A dataset can be very noisy and unpredictable, independently from its size.

To my knowldge, it is not possible to derive any general rule about this.

Answer 2

This question cannot generally be answered, because the training-accuracy is a result of the models' ability to generalize and its ability to overfit.

For example: On one hand, if you decrease the size of your dataset, your training-accuracy might increase, since the model could start learning the random noise of your data (aka. overfit). On the other hand, your training-accuracy could also decrease since the model gets fewer data to learn from and might not generalize as well.