Is there a way to quantify or characterize some upper limits for how much one can "learn" from a dataset?
I got puzzled, when the neural state machine got me thinking components of AGI and lead me to the Visual Genome Dataset and further to generalization in neural networks (although within-distribution limited).
Consider an analogy: an intelligence analyst or X-ray specialist can use their image (satellite or X-ray) and be "informed" by that much better than a layman.
An expert has a) understanding about the domain and b) experience with the images, whereas a neural network only knows b) about the images, not a). Yet, both could make the classification, albeit with different learning material (in practice). The expert can be "informed" by an image and make a judgment of the world, using his/her education, but an image classification system only makes a classification, based on an image dataset.
By the analogy it seems: what can be understood from data is rather a property of the learner / the model, not dataset.
Likewise, a self-supervised network like GPT-2 can learn a great deal about text data, but only when there is huge amounts of it. At the same time, a human can learn a great deal, by reading a single Wikipedia article..
Thus, what can be learned from data depends greatly on the learner.
But, is there a way to characterize, given the learner, what can be learned from a given dataset? I hope this is more than a dataset quality question, maybe some qualitative characterizations could be made.
How should one approach and (properly) understand this question?
(I have feeling this cannot be answered, but let's give it a try).