I want to build a database of image data for machine learning. But how should this be done? I'm assuming people don't just dump all of their image data into a folder? Do they use a relational database management system, like MySQL? Or do they use a NoSQL database, like MongoDB? Is there a textbook that explores this part of machine learning in particular? Is this what "data warehouse" refers to?
4
-
$\begingroup$ This could do with some clarity on scale and scope of the image data store, because different approaches would be taken depending on use cases. So, what is your dataset being used for, broadly? What kinds of annotation do you want to attach to each image? Roughly what size will the dataset be? $\endgroup$– Neil SlaterSep 17, 2020 at 12:03
-
$\begingroup$ @NeilSlater The data will be medical images. There will probably be some image processing that needs to happen at some point, but I'm not doing that immediately, so the images need to be stored first. [...] $\endgroup$– The PointerSep 17, 2020 at 12:33
-
$\begingroup$ [...] At some point, the data will be fed to some machine learning model. The goal is image classification and/or object recognition. Since medical images are difficult to acquire, this will likely be a very small dataset. For supervised learning, there will need to be some descriptor(s), but I'm not yet sure how many/complex these would be. Since they're medical images, I'm assuming the descriptors would probably be considered relatively simple. $\endgroup$– The PointerSep 17, 2020 at 12:36
-
$\begingroup$ I think this depends a lot on the size of your data. As long as your images can be stored on a single folder I don't see what is wrong with this method. You can always have a CSV which lists the paths and the metadata of each image so that you can easily access the images you need during your process. For bigger datasets though I assume you're right but I have no experience in it, so maybe someone else will be able to provide more insights. (but I'd assume that your DB would still be equivalent to the csv, and you'd have some servers acting as folders to download the images from) $\endgroup$– mprouveurSep 17, 2020 at 13:05
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
There are several approaches to this as you need both the input (images) and if your problem is a classification one, you need to reliably store the labels. You might also have some additional information about the images that could be useful for your problem:
you can store the images in such a way that all information is contained in the permanent store (for instance folder names with the labels that you want to learn and all the images of a given class within that folder). Keras has a method that allows you to create a dataset from a directory
tf.keras.preprocessing.image_dataset_from_directory.another way (which I prefer) is to store in a (SQL) database all of the metadata (label, image url in a table for instance). This is more flexible because you can easily change a label, add a new category without having to move images around. This also allows you to change the format and add additional data related to each image.
-
$\begingroup$ What do you mean by "image url"? These images are being stored on our machines. $\endgroup$ Sep 17, 2020 at 13:33
-
$\begingroup$ yes: if your images are local then image url = local path but usually for large enough dataset the images would be on some distant bucket (then you need the url to download them) $\endgroup$ Sep 17, 2020 at 13:35
-