# How to build a database of image data for machine learning?

I want to build a database of image data for machine learning. But how should this be done? I'm assuming people don't just dump all of their image data into a folder? Do they use a relational database management system, like MySQL? Or do they use a NoSQL database, like MongoDB? Is there a textbook that explores this part of machine learning in particular? Is this what "data warehouse" refers to?

• This could do with some clarity on scale and scope of the image data store, because different approaches would be taken depending on use cases. So, what is your dataset being used for, broadly? What kinds of annotation do you want to attach to each image? Roughly what size will the dataset be? Sep 17 '20 at 12:03
• @NeilSlater The data will be medical images. There will probably be some image processing that needs to happen at some point, but I'm not doing that immediately, so the images need to be stored first. [...] Sep 17 '20 at 12:33
• [...] At some point, the data will be fed to some machine learning model. The goal is image classification and/or object recognition. Since medical images are difficult to acquire, this will likely be a very small dataset. For supervised learning, there will need to be some descriptor(s), but I'm not yet sure how many/complex these would be. Since they're medical images, I'm assuming the descriptors would probably be considered relatively simple. Sep 17 '20 at 12:36
• I think this depends a lot on the size of your data. As long as your images can be stored on a single folder I don't see what is wrong with this method. You can always have a CSV which lists the paths and the metadata of each image so that you can easily access the images you need during your process. For bigger datasets though I assume you're right but I have no experience in it, so maybe someone else will be able to provide more insights. (but I'd assume that your DB would still be equivalent to the csv, and you'd have some servers acting as folders to download the images from) Sep 17 '20 at 13:05

• you can store the images in such a way that all information is contained in the permanent store (for instance folder names with the labels that you want to learn and all the images of a given class within that folder). Keras has a method that allows you to create a dataset from a directory tf.keras.preprocessing.image_dataset_from_directory.