Creating a custom dataset for object detection

Question

I'm currently trying to build a model to recognize approximately 10 labels (food items) in a fairly controlled environment (refrigerator). I was unable to find datasets that worked well enough for my task, so I'm attempting to curate one myself.

So far this has been my approach:

• Capture images with a smartphone
• Annotate in LabelImg
• Train in Detectron2 / pyTorch

I have 2 questions:

1. Given that the environment is somewhat consistent throughout all samples, is there a ballpark number / rule of thumb for a decent number of samples to use per class? (100 train / 100 test)? This is a proof of concept project, so I'm just looking for something with reasonable accuracy (80%+)

2. After I've captured my images (let's say via smartphone) are there any preprocessing steps (other than annotating) that are necessary before using as training data? (i.e resize the images, reduce file size, format)

Absolutely any help is appreciated, and of course other tips/suggestions you feel may be useful.

Answer 1

I have built many data sets. The latest was a data set of species of birds. I has 100 species of birds so I had 100 classes. For each species (class) I had 100 training images, 5 test images and 5 validation images. Thats as total of 11,000 images.. The classifier I built had a final accuracy of 98% on 500 test images (5 test images per specie) Here are some things to consider in building your data set. 1- crop the data or not- depending on the images you generated via your cell phone what is on average the ratio of the pixels of the region of interest (example a milk bottle in the refrigerator) to the total pixels in the image. Your classifier will always work better when this ratio is high. You want your classifier to learn to recognize a milk bottle as a class.Any part of the image that is not a milk bottle is essentially noise. The rule of thumb I use to build a high quality data sets is 50%. That is that on average for your data set the region of interest should occupy 50% of the pixels in the image. 2- how many training- validation-test images should you have? depends on the problem but if you follow the 50% rule above I think you can build a solid data set with about 100 images per class, 5 validation images per class and 5 test images per class so you would need about 110 total images per class. 3- image augmentation - even with the small number of images you can create a good classifier using image augmentation. This is a way of artificially expanding the size of your data set. If you use Keras, the ImageDataGenerator provides several transforms that can be used to augment your data set. Documentation is at https://keras.io/preprocessing/image/. 4-image size - CNN's operate with a fixed size for all input images. Again in general the larger the image the better the result(assuming the 50% rule is used). However you pay a price for large images in terms of computation time and memory usage. I have consistently used 224 X 224 X 3 (color images). Depending on the importance of color(ie how important is color to discriminating between classes) you may be able to just use gray scale images 224 X 224 X 1.

5- avoiding BIAS in your data set - make it diverse as possible. It is easy to accidentally build an unintentioned BIAS in your data set. Here is an example. A guy created a data set with 2 classes. One set had images was of various kinds of dogs. The other set had only images of wolves. He wanted to build a classifier that if given an image would classify it as either a dog or a wolf. He built his data set and trained the classifier with great results 99% training accuracy 98% validation accuracy. That is a surprising result since a lot of dogs very closely resemble a wolf. But when random images of a dog or a wolf were presented the accuracy was essentially 50%. WHy? Well he built a big bias in his data set. 90% of the training and validation images for wolves had a white background of snow. Since he did not crop the images most of the image was white. For the dogs his images were mostly of dogs NOT in snow. So what the CNN learned was "if background is white must be a wolf, if background is not white must be a dog. So think about avoiding building an unintentional bias in your data set. In your case say you have a class ketchup bottle. Now if all you take are images of HEINZ ketchup bottles (probably with the text HEINZ on the label) your model will not know anything about ketchup bottles that aren't Heinz ketchup bottles. Good luck I hope your task works out well. One more thing, why are you considering doing all the work of taking cell phone images? Why not just do a Google search for the items of interest and download the images. For example do a search for "ketchup bottle images" You will find hundreds of images (with diversity which is important) that you can just download.

Answer 2

Given that the environment is somewhat consistent throughout all samples, is there a ballpark number / rule of thumb for a decent number of samples to use per class? (100 train / 100 test)? This is a proof of concept project, so I'm just looking for something with reasonable accuracy (80%+)

This really differs from problem to problem. The number of observations needed to reach a given threshold of quality depends on how easy it is for a Neural Network to classify them correctly. Unfortunately, only a person with your specific domain knowledge can say that. I have seen CNN multi-classification tasks made on few hundres of observations per class. In that case, a massive amount of data agumentation would be fundamental. A good rule of thumb is to be at least in the order of thousands. However, hundreds of obs. per class + data augmentation might work, I suggest you to try with a smaller dataset, and increase its size in case it's not enough.

---

After I've captured my images (let's say via smartphone) are there any preprocessing steps (other than annotating) that are necessary before using as training data? (i.e resize the images, reduce file size, format)

Image size is the most important issue. CNNs require an input of constant size (height, width, channels). Conv layers can already take care of zero padding of smaller images. However, you might rescale larger ones. You can create an input pipeline to preprocess image data, that can then be fed into the CNN. The main purpose of this pipeline is to keep the images in an acceptable size (depending largely on you computational power).