I'm looking for a research project for datasets (the more the better), potentially crowdsourced, which go beyond basic feature vectors + labels, and include additionally some metadata about the labels. Specifically I'd like the annotation time, or some other cost metric, and if the dataset was crowdsourced, the individual labels per annotator. On top, some measure of the quality of the labels by different annotators would be helpful as well. The Dataset domain is not relevant.

Also I'm not sure if https://datascience.stackexchange.com is the right place to ask for that, but maybe you can help me.

So far I could only find this dataset.

  • 1
    $\begingroup$ This question may be a better for opendata.stackexchange.com $\endgroup$ – noe Feb 1 at 11:01
  • $\begingroup$ Oh thanks, didn't know that one existed! $\endgroup$ – Ragadabing Feb 1 at 11:05

I did a project with the CheXpert dataset which includes metadata and labels for uncertainty. I don't know if it's the right kind of metadata you are looking for. In general, medical datasets usually contain metadata. A similar one would be from the NIH.

Otherwise, this is a decent index of high quality datasets.

  • $\begingroup$ No, CheXpert is not what I'm looking for. Apparently the used multiple experts for the labels of the validation set, but not even the individual labels per annotator are provided, metadata like the duration of how long it took the annotator to label the samples. Same applies to the NIH one. $\endgroup$ – Ragadabing Feb 5 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.