Classifying objects in video without machine learning

Question

Recently, Nick Bourdakos posted a series of videos demonstrating bottle detection in a video stream using Tensorflow.js. Specifically, he is using SSD-mobilenet.

The problem could be summarised as follows:

• Three different drinks bottles appear together or individually in a video stream. Let's assume Coke, Mountain Dew and Pepsi as in the original video
• Each bottle must be classified and labelled with its name
• A rectangular bounding box should be placed around each bottle

I am interested to know if there are there any established techniques that can achieve similar results without machine learning?

So far I have tried:

1. Identification based on colour thresholding. This works surprisingly well but is not very robust. For example the red lettering inside the Mountain Dew label can be falsely detected as "Coke". Also, bounding boxes can only be drawn around coloured regions rather than the entire bottle
2. Template based matching. This didn't work at all. I assume this is intended for static images where an exact match is required.
3. KCF Tracking. This seems to work well, but I have to manually define a region of interest to track first. That is, objects are tracked but not classified
4. I also tried using edge detection to classify "bottle" or "not bottle" based on the aspect ratio of edged objects, however because the bottles are "hand held" separating the bottle edge from the user's hand is problematic

Is there some computer vision technique or steps that I'm missing that would come closer to solving the problem?

Answer 1

You could try SIFT (Scale-invariant feature transform) to recognize the labels on the bottles. Althoug you need some preprocessing, just for comparing the labels on the bottle and to distinguish the different brands it should work. Also it could get kind of complex (if you show the bottles from behind and the brandlogo is not visible). So you maybe have to extract features from many perspectives of the bottle.

Here is one example: https://towardsdatascience.com/bibirra-beer-label-recognition-8546c233d6f4