How to detect cardboard boxes using Neural Network

Question

I'm trying to train a Neural Network how to detect cardboard boxes along with multiple classes of persons (people).

Although it's easy to detect persons and correctly classifies them, it's incredibly hard to detect cardboard boxes.

The boxes look like this:

My suspicion is that box is too simple of an object, and the neural network has a hard time detecting it because there are too little features to extract from the object.

The division of the dataset looks like this:

personA: 1160
personB: 1651
personC: 2136
person: 1959
box: 2798

Persons are wearing different safety items, based on the items are classified, while detected as whole person, not just the item.

I tried to use:

ssd300_incetpionv2
ssd512_inceptionv2
faster_rcnn_inceptionv2

All of these are detecting and classifying persons much better than boxes. I cannot provide exact mAP (don't have it).

Any ideas?

Thanks.

Answer 1

As you said:

My suspicion is that box is too simple of an object, and the neural network has a hard time detecting it because there are too little features to extract from the object.

... and that is exactly the problem with this task. I suggest you to train your Network using a whole lot of image augmentation. If you are using Keras/TensorFlow 2.0, they have built-in functions that do that.

I also suggest you to train a first model, then study accurately the images that your Network thinks are boxes but aren't (the false positives). At that point, you select the "correct" cases, together with the false positive cases, and build a subset of the dataset that you have. That sub-dataset can be useful to train the Network to distinguish between actual boxes and objects that look like boxes.

Once again: use a massive load of image augmentation. That's my main point, and it's what I would do.

Answer 2

I suggest using a pretrained model.

Here is the full code of a pretrained multiclass image classification I did recently: https://datascience.stackexchange.com/a/52772/71442

There are different pretrained models you can try: https://keras.io/applications/

With pretrained models, you can „reuse“ the convolutional layers of large models and train your classes on top. This may help to find the specific features associated with the boxes.

Boxes should be easy to detect because of the distinct and uniform shape and color of them. So I think trying a pretrained model which is trained on similar (box-like) classes can help.

If all the boxes are brown (like in the example image), you may even be able to detect boxes based on the color patterns. NN is able to do that.

Overall it is hard to say what did go wrong without seeing your model. The model architecture can be relevant here.

Answer 3

You can try using existing TensorFlow models that have been pre-trained on a large datasets such as MS-COCO, Kitti, and Open Images etc. You can then fine tune a particular model that you wish on your dataset.

A list of all available models to use can be found here.

A good source on how to set up an object detector. (Don't worry if you are Linux user, the same tutorial can be adapted even in Linux systems).

You can use a tool like labellmg to label your images for training.

Hope this helps.