In my daily machine learning / deep learning workflow, I often want to interact with a dataset in my code. Specifically, I would like to be able to load some module / package which can

  1. make sure that I have the dataset in a specific folder
  2. represent the structure of the dataset on a higher, possible object oriented level
  3. is homogeneous across datasets of the same class, e.g. image classification (set of images with labels, possibly bounding boxes)
  4. open source, such that I could contribute a specification to a previously uncovered dataset
  5. that let's me specify a set of commonly used licenses to which I can comply, to filter all compatible datasets

Currently I mainly use Python, but other languages are also in the scope of this question (Matlab, Java). Does such an API exist? If not, which ones come closest to the requirements stated above?

To give you an example of how I would expect it to feel like, see the following python code

import dataset_api as da

in_2012 = da.get('ImageNet', version='2012')
in_2017 = da.get('ImageNet', version='2017')
coco = da.get('COCO')

image_classification = da.merge([in_2012, in_2017, coco])

images = image_classification.images

image = images[0]
image.path  # the absolute file path to the image file on my disk
image.objects  # an array of object that are visible in this image

Note: this question is also posted on ai.stackexchange.com. I wasn't sure which exchange would better fit the purpose.

  • is it like sodapy? – bakka Sep 25 at 23:56
  • cool, thanks for pointing this out! I would expect a similar API. However to me it seems that Socrata offers more atomic data in the form of databases (that are queried). I am specifically looking for APIs of common machine learning datasets that are part of challenges – maddin25 Sep 26 at 12:07

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.