Neural networks - Find most similar images

Question

I am working with Python, scikit-learn and keras. I have 3000 thousands images of front-faced watches like the following ones: Watch_1, Watch_2, Watch_3.

I want to write a program which receives as an input a photo of a real watch which maybe taken under less ideal conditions than the photos above (different background colour, darker lightning etc) and find the most similar watches among the 3000 ones to it. By similarity I mean that if I give as an input a photo of a round, brown watch with thin lace then I expect as an output watches of round shape, of dark colour and with thin lace.

What is the most efficient machine learning algorithm to do this?

For example, by following this link I have two different solutions in my mind:

1) Using a CNN as a feature extractor and compare the distances between the these features for every pair of images with reference to the input image.

2) Using two CNNs in a Siamese Neural Network to compare the images.

Are these two options the best ones for this task or would you suggest something else?

Do you know any pre-trained neural network (with pre-determined hyperparameters) for this task?

I have found some interesting posts on StackOverflow about this but they are pretty old: Post_1, Post_2, Post_3.

Answer 1

I don't think that a high level architecture as such is the best fit but it rather depends on many factors and details. From what I am aware of the first approach is promising especially when extended by additional steps as done in the TiefVision:

1. An additional bounding box network is used to discriminate the relevant part of the image from the rest
2. The feature vectors are not simply compared directly but rather used to train a comparison network using triplets (learning similarity based on feature vectors and examples for more and less similar instances).

This work is a lot more recent (2016/17) than what you posted and comes with a nice tool set and a more detailed paper.

Why using triplets aka Deep Ranking?

As stated in the comments: Why should one use triplets for image similarity instead of learning feature vectors and calculate their distance? Triplets are a way of formulating the similarity question as a learning problem instead of learning feature vectors that basically do not care for similarity. This approach makes especially sense in cases where human-perceived similarity is important, which might differ from machine perception.

Triplets work like this: You provide 3 images. One to compare to, one similar (close) and one not so similar (distant) image. This is your training/test/validation data. Training your network on those samples and predicting the correct order (classify similar from non-similar images) overall lets the network learn how to order images based on their similarity.

All in all this approach is comparatively complex. It might be overengineered but you also asked for the best way to do this and Deep Ranking achieves very high precision values.

Answer 2

I would pick a classifier, like VGG-16, that works well on the imagenet classes. Then, run your watch images through it. For sure, you can expect the output to be mostly "watch" with high probability.

However, you then get extra features: the activation level of all other categories. That gives you a vector of a thousand values between 0 and 1.

You can also extract the activation at various point in the network. Then, the similarity of those activations and outputs should be similar between two cases only if the images are similar.

Answer 3

I would focus on data augmentation first. Since your images have a white background you have it a little bit easier. Turn the white background into a transparent background, scale down the image, rotate it and put it in backgrounds similar to your target data.

Do this a bunch of times with different combination and have a label for each watch. Then I would suggest you use a regular convolutional neural network for the classification. Each label will have a score, pick the one with the highest confidence and that one should be the most similar.

For example lets say you run the classifier with an image and get this result:

Watch1: 0.51

Watch2: 0.30

Watch3: 0.25

The CNN is saying that it has a 51% confidence that Watch1 is the watch in the input image. But also what is true is that it is the one it thinks looks more similar, Watch2 would be the next one more similar and so on.

If you don't get good results, do the usual. Experiment with the parameters and/or add more layers. Try to find out where it is failing. After you have that insight, you can use it to choose a more specialized type of convolutional network for your particular problem. Looking for that without prior knowledge of how it would perform is not the right approach. I would suggest you start with a basic convolutional model and then work from there.

Answer 4

I would try to train a neural network with some sort of self-supervised approach, where you take all of your images and you change them in some ways (mess with colors a bit, rotate, rescale, etc.) and the task of the network is to create embeddings for these two to be close together and far from all the other images.

The network will probably have harder time to push away more similar images than dissimilar ones.

One such self-supervised approach is SimCLR for example. Could worth a try.