why does performance of machine learning models plateau after certain amount of training data

Question

I'm referring to the below image which I came across.

The explanation seemed intuitive at first but I don't think I understand how it works. The image says that the performance of traditional machine learning algorithms plateau after a certain amount of data while that of deep learning algorithms gets better with more data. Does it have to with the feature learning where deep learning methods automatically learn the important features as opposed to the manual feature selection for machine learning models? If so, can you please elaborate on that.

Answer 1

In part feature learning has a lot to do with the fact that deep learning models can learn more data distributions. Although the graph that you are showing isn't completely true. For instance you can see the No Free Lunch Theorem for machine learning, that states that no machine learning model is universally any better than any other.

Either way, the goal of machine learning is not to seek a universal learning algorithm. Instead our goal is to understand what kind of dat distributions are relevant to the the real world that an AI agent experiences, and what kinds of machine learning algorithms performs well on data drawn from the kinds of data generating distributions we care about.

In conclusion, when facing a machine learning problem you should try to find the model that best learns the kind of data you are trying to work with and this graph is poorly informative and has no foundations.

Answer 2

One reason that the older learning algorithms did not perform very well was that the algorithms were not that much powerful and deep to be able to extract different features in the images. It was not possible to use a very deep neural network because of the problems such as vanishing gradients. However, now, there are different methods (e.g., ResNet concept in case of CNN, LSTM in case of RNN) and also different nonlinearity functions (e.g., Relu) that could be useful to prevent overfitting. Now because of these improvements, we are able to use very deep neural networks. There is a still an issue, and that is overfitting. When, the neural network is very deep, the chance of the overfitting is very high. The solution for that is to use a large dataset.

In summary, using a large dataset now could be useful to improve the performance of the training dataset because very deep neural networks are able to detect and extarct different features of data. On the other hand, using a very large dataset set could also prevent from overfitting.