# Neural networks - Find most similar images

I am working with Python, scikit-learn and keras. I have 3000 thousands images of front-faced watches like the following ones: Watch_1, Watch_2, Watch_3.

I want to write a program which receives as an input a photo of a real watch which maybe taken under less ideal conditions than the photos above (different background colour, darker lightning etc) and find the most similar watches among the 3000 ones to it. By similarity I mean that if I give as an input a photo of a round, brown watch with thin lace then I expect as an output watches of round shape, of dark colour and with thin lace.

What is the most efficient machine learning algorithm to do this?

For example, by following this link I have two different solutions in my mind:

1) Using a CNN as a feature extractor and compare the distances between the these features for every pair of images with reference to the input image.

2) Using two CNNs in a Siamese Neural Network to compare the images.

Are these two options the best ones for this task or would you suggest something else?

Do you know any pre-trained neural network (with pre-determined hyperparameters) for this task?

I have found some interesting posts on StackOverflow about this but they are pretty old: Post_1, Post_2, Post_3.

• Welcome to the site! Learn an embedding (I recommend a binary hash for fast retrieval on the order of a thousand bits) then perform similarity search. – Emre Feb 14 '18 at 20:52
• It's my pleasure to be here...haha!...Yes, this is a good idea and I had already read this paper... – Outcast Feb 15 '18 at 10:43

I don't think that a high level architecture as such is the best fit but it rather depends on many factors and details. From what I am aware of the first approach is promising especially when extended by additional steps as done in the TiefVision:

1. An additional bounding box network is used to discriminate the relevant part of the image from the rest
2. The feature vectors are not simply compared directly but rather used to train a comparison network using triplets (learning similarity based on feature vectors and examples for more and less similar instances).

This work is a lot more recent (2016/17) than what you posted and comes with a nice tool set and a more detailed paper.

Why using triplets aka Deep Ranking?

As stated in the comments: Why should one use triplets for image similarity instead of learning feature vectors and calculate their distance? Triplets are a way of formulating the similarity question as a learning problem instead of learning feature vectors that basically do not care for similarity. This approach makes especially sense in cases where human-perceived similarity is important, which might differ from machine perception.

Triplets work like this: You provide 3 images. One to compare to, one similar (close) and one not so similar (distant) image. This is your training/test/validation data. Training your network on those samples and predicting the correct order (classify similar from non-similar images) overall lets the network learn how to order images based on their similarity.

All in all this approach is comparatively complex. It might be overengineered but you also asked for the best way to do this and Deep Ranking achieves very high precision values.

• Thank you for your response. This is an interesting one (I upvoted it). The idea about the triplet is good even though I am not exactly sure why using triplets of images is necessarily better than using pairs of images for my task. If you want to, you may explain it more at your post. Also I will have a look at TiefVision. – Outcast Feb 14 '18 at 14:41
• @Universalis thanks for the hint (and the upvote). I updated my answer with some more details about triplets and the reasoning. The paper is also written very well so have a look for all the details. There might be new fancy ways to do this since TiefVision and DeepRanking came up, though. – Gegenwind Feb 14 '18 at 16:25
• Hmm maybe I misunderstood your approach. The triplet provides comparative information that image 1 is closer to image 2 than to image 3. Having only 2 images and stating "these 2 are similar" lacks a comparative factor "similar in what respect" because in this approach you assume that the plain distance of the feature vector does not tell you enough. in other words: You learn similarity by order and you lack that without having at least 2 items to order. – Gegenwind Feb 14 '18 at 17:41
• Thank you again for your response. However, even now and after reading a bit more carefully the paper, it is not clear to me why you necessarily need triplets and not pairs of image for this unsupervised approach. When using labels it is clear that by using triplets you will get a full similarity ranking of the images that you cannot get with pairs. But in this (sort of) unsupervised approach which you propose then why not to compare the values of the loss function for each pair of images (always one image of the pair is the input image) to find e.g. the 5 most similar ones to the input image? – Outcast Feb 15 '18 at 12:05

I would pick a classifier, like VGG-16, that works well on the imagenet classes. Then, run your watch images through it. For sure, you can expect the output to be mostly "watch" with high probability.

However, you then get extra features: the activation level of all other categories. That gives you a vector of a thousand values between 0 and 1.

You can also extract the activation at various point in the network. Then, the similarity of those activations and outputs should be similar between two cases only if the images are similar.

• Thank you for your response (upvote). Yes, I had this in my mind and in a sense it is related to the first option which I provided at my post. So I was also thinking about using other detectors like SURF...I will see if these are sufficiently successful... – Outcast Feb 15 '18 at 10:08

I would focus on data augmentation first. Since your images have a white background you have it a little bit easier. Turn the white background into a transparent background, scale down the image, rotate it and put it in backgrounds similar to your target data.

Do this a bunch of times with different combination and have a label for each watch. Then I would suggest you use a regular convolutional neural network for the classification. Each label will have a score, pick the one with the highest confidence and that one should be the most similar.

For example lets say you run the classifier with an image and get this result:

Watch1: 0.51

Watch2: 0.30

Watch3: 0.25


The CNN is saying that it has a 51% confidence that Watch1 is the watch in the input image. But also what is true is that it is the one it thinks looks more similar, Watch2 would be the next one more similar and so on.

If you don't get good results, do the usual. Experiment with the parameters and/or add more layers. Try to find out where it is failing. After you have that insight, you can use it to choose a more specialized type of convolutional network for your particular problem. Looking for that without prior knowledge of how it would perform is not the right approach. I would suggest you start with a basic convolutional model and then work from there.

• Thank you for your response (upvote). Yes, I had already data augmentation in my mind. However, your response is quite unclear. What do you mean by "have a label for each watch"? Do you mean labelling each watch individually or labelling them as a pair with another one depending on whether they are similar or not? (If it is the former then explain why this is efficient please) – Outcast Feb 14 '18 at 16:50
• It is the former suggestion, I was assuming you already had a handy way to label all of them. This is basically the most basic solution so it won't be efficient. My suggestion was that the basic model could perhaps give you enough information for you to chase a more specialized model. Looking a deep ranking seems very promising like @Gegenwind said. I have updated my answer to make it a little more clear. – zimio Feb 14 '18 at 20:17
• Hm, yes now it is more clear what you meant....This is certainly a good general suggestion: start with the basics and move on step-by-step to more complex stuff...Thanks in any case... – Outcast Feb 15 '18 at 10:00

I would try to train a neural network with some sort of self-supervised approach, where you take all of your images and you change them in some ways (mess with colors a bit, rotate, rescale, etc.) and the task of the network is to create embeddings for these two to be close together and far from all the other images.

The network will probably have harder time to push away more similar images than dissimilar ones.

One such self-supervised approach is SimCLR for example. Could worth a try.