Is it proven that all 15M images were manually classified correctly and there are no mistakes or randomly selected responses collected?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Image classifiers today are actually used to find the instances of bad/ wrong labelling in the data bases that where used to train them. Essentially you train a neural network on some data set until you get a very high success rate. Then have a human being look at the images that where misclassified. This can give you some understanding on how to improve the neural network but often it also just turns up the instances where images are badly/ wrongly labelled. $\endgroup$– quaragueCommented Sep 6, 2022 at 6:44
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
0
It was actually shown that imagenet and many other famous datasets have lot's of errors in them, including bad labels. See this nice post on the topic:
https://medium.com/@amiralush/large-image-datasets-today-are-a-mess-e3ea4c9e8d22
$\begingroup$
$\endgroup$
1
In addition to what Iyar Lin has said, I'll mention that there is no such thing as a large organic image dataset with no errors. Even if we were to get many intelligent humans to analyze each image, lots of the images would have too much disagreement to say what is the "correct" label.
I and another person wrote a report analyzing a small part of the OpenImages dataset. Here is a bit of what we learned. (Full disclosure, this includes some quotes and paraphrasing of that report, but I cannot cite it as it is not public.)
Look at this image. It appears to feature prosciutto, arugula, and a sprinkling of cheese on top of a toasted base. Is it pizza? It is clear what is contained in the photo, but your answer depends on how you define pizza. You can ask several people, and you will get answers ranging from "probably pizza" to "definitely not pizza" and everywhere in between. No matter whether you label this as a pizza or not, there will be people who say you're wrong.
Now consider this. It is a pizza box with a picture of a pizza on it. Is it pizza? Again, it's clear what is in the photo, but there's no right answer to what counts as a pizza, since it depends on context. It is clearly a picture of a picture of pizza, but you sometimes might want to count it as pizza, and sometimes might not.
Lastly, consider this dish. Unlike the past two photos, it is unclear what's in the photo. There's definitely eggs on top, but underneath there are things that could be dough, cheese, and pizza sauce, or could be scrambled eggs and ketchup. Assuming for the sake of the argument that pizza can have fried eggs on top, whether or not this counts as pizza cannot be determined from the photo.
To conclude: while it may be possible to make a "100% correct" dataset by discarding ambiguous samples, it would no longer represent what's in the real world. In the real world, things are ambiguous, and there are "errors" that will always exist due to the philosophical ambiguity of our environment.
answered Sep 6, 2022 at 23:30
-
2$\begingroup$ Excellent points. It's pretty easy to get very good results with digits recognition, e.g. from MNIST database. It's not possible to get 100% correct results, simply because some digits are so badly written, and it isn't possible for humans or computers to read if it's supposed to be 3, 8, 5 or 9. BTW, the last dish could be en.wikipedia.org/wiki/Shakshouka $\endgroup$ Commented Sep 7, 2022 at 14:19