Preprocess image data to classify objects based on shape

Question

Currently I'm trying to build a neural network that is able to classify different types of bottles on an image solely based on the shape. The bottles have no label and at first I only used beer and wine bottles. My goal is to compare a neural network that uses feature extraction to one that uses the raw images.

Example input image:

https://i.gyazo.com/c22151a4bee6146cc689fcb8358483dd.jpg

Using openCV to process the images, I tried to obtain only the edges of the bottles and classify the types based on the shape. Currently, I am receiving the following result for the images:

https://i.gyazo.com/8554b06ebf6e8bd520d11222f70c77c5.png

In the next step, I cropped out each bottle in the image and split them into a 2d array of tiles (normally I try with 30x50 but in this image I show it with 10x20)

https://i.gyazo.com/28fb1bc0fa8d61b185ed2245db222cc8.png

For every tile I execute PCA to extract the eigenvectors. In the end I receive a matrix of 10x20 tiles, where each of them contains the direction of the line. As you can see, the (0.0, 1.0) tuples are the vertical lines, and the (1.0, 0.0) tuples are the horizontal lines.

https://i.gyazo.com/bcd291535ea01b6392ab6aef6747de9d.png

Sorry for the long post, here is my question:

Is it possible to use this data do classify between wine and beer bottles? The shape of the neck of the bottle is different but is it representative for building a neural network? I tried to train a model by calculating the gradient (arctan from the tuple) but i didn't manage to get some successful results. Does anyone have a idea how to pre-process the data for a neural network?

Is it better to use the cropped out images of the bottles as image data for the neural network?

I am thankful for any help as this is my first machine learning project and I would really like to get it working.

Answer 1

Well, you are trying so describe the image by predominant line directions and that really sounds fair but a bit of too much work. If you want to you can try a bit of data augmentation to increase your numbers of samples (Mainly affine transformations)

Other possible useful feature would be descriptors such as:

• SIFT
• SURF
• FAST
• BRIEF

You could try to describe those shapes using other geometrical functions such as skewness (fill the inside of the shape, consider the eigenvalues to define the major axis directions on PCA)

Finally, bottles in the sample image look similar in shape and differ mainly in size, if your model uses resize it might become scale invariant and then your method will decrease too much of your shape information and give bad results.

------------------------------------------------------------------------------------

Answering the comment bellow:

This is why i tried to do it with my approach of cropping out each bottle and split them into a 2d array of tiles (e.g. 50x20)

This inserts a scale invariance to your model, that is good for processes where image-size vary either because of distance or camera resolution but: This throws alway scale information, that might not be helpful when the main discriminant on your image is scale-related.