Anyone have pointers to where the human level performance on ImageNet comes from?
I found a reference to 5.1% accuracy (top-1? or top-5?) from here.
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
It comes from this paper: https://arxiv.org/abs/1409.0575
O. Russakovsky "ImageNet Large Scale Visual Recognition Challenge" 2014
$\begingroup$
$\endgroup$
TL;DR: 5.1% top-5 classification error (expert #1), but 2.4% optimistically.
In the paper ImageNet Large Scale Visual Recognition Challenge by Russakovsky et at. (2014), there is a section in which they estimated the human classification error for ILSVRC (Section 6.4.1 Quantitative comparison of human and computer accuracy on large-scale image classification).
They used two human experts to annotate images from the test set. The first annotator (A1) evaluated 1500 images and obtained a top-5 classification error of 5.1% ("trained" with 500 images). Then, the second annotator (A2) evaluated 258 images with a top-5 classification error of 12.0% ("trained" with 100 images).
Furthermore, the authors approximated the error rate of an "optimistic" human annotator at 2.4% (top-5).
ps.: It is not an easy task for a human being to classify an image when there are a thousand possible classes. In addition to this, it must be taken into account that there is a considerable overlap between classes for some images. Try for yourself here
