When reading about deep learning I often come across the rule that deep learning is only effective when you have large amounts of data at your disposal. These statements are generally accompanied by a figure such as this:
The example (taken from https://hackernoon.com/%EF%B8%8F-big-challenge-in-deep-learning-training-data-31a88b97b282 ) is attributed to a 'famous slide from Andrew Ng'. Does anyone know what this figure is actually based upon? Is there any research that backs up this claim?
The original slide in question "Scale drives deep learning progress" is possibly what you currently can find at https://cs230.stanford.edu/files/C1M1.pdf (page 13). It may be roughly interpreted with "low bias learners [in that plot, larger neural networks] tend to benefit from more training examples".
The main reason is that in deep learning the number of training parameters are so many and there is a fact that for each parameter you need at least $5$ to $10$ data to have a good prediction. The reason is a bit complicated to explain but it is related to pack learning and if you insist to know why, I can tell you that in the error term for the test data, you have an overfit term which grows with the number of sample size if your training model is a kind of hypothesis that increases when the number of data increases. In hypothesis with the growth of $O(2^n)$ it is impossible to make the generalisation error same as training error, such as 1NN on the contrary, hypothesis with the growth $O(n^c)$ which are limited to polynomials can have an overfit which can be diminished by increasing the size of training data. Consequently, if you increase the size of your data you can have better generalisation error. Deep learning models obey the second growth manner. The more data you have, the better generalisation you have.
deep learning does not require feature reduction to improve accuracy. deep learning automatically inhibits features that do not contribute to the accuracy in the outcome. The big data represents rules or functions in the data called signal. The more signal that can be discovery by the deep learning network the greater its accuracy.
Not many people have looked into this problem from a theoretical perspective. There's a line of research that focuses on model selection that might be worth considering.
Model selection is intertwined with data in stochastic learning. Traditionally, there are two main elements that dictate which model should be used given the data, the type of model and complexity. In learning, data drives the model component, while complexity is dictate by the how it is measured and the model architecture. This is all an elegant way of pointing out what is essentially equivalent to the bias-variance trade-off. Therefore, the amount of data that is required typically saturates at some point in the training process and that is inherently because of the bias-variance trade-off principle and its dependence on the complexity of the model.
One of the observations in deep learning has been that adding more data doesn't seem to saturate. This led some people to revisit the notion of bias-variance trade-off. In fact, it turns out that for deep learning models, it might be true that our traditional view of the bias-variance trade-off no longer holds. I encourage you to read [1] for a complete description, as well as [2].
Rather than questioning deep learning as some extraordinary class of model that is somehow data-hungry. It might be good to think about the limitations of our understanding of model selection and its relationship to data.
[1] Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116(32), 15849-15854.
[2] Dwivedi, R., Singh, C., Yu, B., & Wainwright, M. J. (2020). Revisiting complexity and the bias-variance tradeoff. arXiv preprint arXiv:2006.10189.
