2
$\begingroup$

I know what content based image retireval is. I have read this and this as one of them says: "given a query images, get a rank list that are most similar to the query image, based on the content of the query image. " But my question is how the "similar" images are determined. Assume we are working on Oxford5k dataset. The dataset contains 5k images in 17 classes. So, when I feed one of the images as a query, my algorithm returns a list of ranked images. images come in the top of this list consider similar to the query. I know that we do the retrieval by extracting feature and matching them. But, how I can evaluate whether my method finds better than the others.

I am pretty sure that some would say an image is similar to another one provided that they both belong to the same class. But this means we are doing a classification not an image retrieval task. If it is not the case, how I can say method A finds more "similar" images than method B.

I am aware about Precision, Recall and mAP. But my question is before these metrics. For example, precision is TP/(TP+FP). But how we determine one image is a true positive (just based on class names?)

How we say an image is similar to another one? Considering them as similar images based on belonging to the same class does not make sense.

Any clear explanation would be really appreciated.

UPDATE 1: My question is actually a question about the literature. How computer scientist working on field of image retrieval, consider an image similar to another one. How they measure their methods and claim theirs are better than the state-of-the-art? using precision metric only? Anyone can interpret as he likes to make his method looks better. For example, I can say an image of white bird is similar to an image of airplane as both have wings and white. And this is a correct, they are similar. By this assumption, I can say my method is better than yours. It means there is no standard benchmark. Really? after more than two decades progress in image retrieval, we have no solid standard to measure similarity and comparison technique to compare two method? just a mAP score which I can argue that anyone can cheat and fabricate better results by playing with the ranks.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

There is no single best similarity metric, unless a query and found images are near identical.

Similarity is not a universal concept. It is trained. Maybe it happened to you to say that a person A looks like B, but your friend does not see the similarity.

Here is a nice example of image similarity by Flickr Code illustrating the concept. This article collection helps better understand similarity. Here is a nice example of practical solution to find image similarity for fake photos.

I am closely monitoring the topic on Stackexchange, only to observe that many people are trying in vain to find some magic metric. A typical beginner uses SSIM for everything. But the problem is intractable, because of huge number of visual information dimensions, especially semantic ones. One could only imagine what image similarity is for an insect or an octopus.

Deep learning works, because a trained network is an approximation and generalization over training examples. It is like web-search results, which can be good based on click stats from many users, but not good in one specific case.

It is a ranking problem with focus on particular metrics of interest, or common cases.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I agree with you. But my question is actually a question about the literature. How computer scientist working on field of image retrieval, consider an image similar to another one. How they measure their methods and claim theirs are better than the state-of-the-art? Anyone can interpret as he likes to make the method looks better. For example, I can say an image of bird is similar to an image of airplane as both have wings. By this assumption, I can say my method is better than yours. $\endgroup$
    – David
    May 11, 2022 at 17:43
  • $\begingroup$ Computer scientists use labelled datasets or set up labelling tasks themselves and ask humans to label similar images, or determine the class of each image. Each result will depend on subjective relation between the task and the human labeller (for example, their culture). There are also some interesting unsupervised methods which define similarity of objects as their proximity in space or time, like vision transformers or word2vec/transformers for text. $\endgroup$
    – Valentas
    May 13, 2022 at 8:29
  • $\begingroup$ @Valentas, I understand your point. But, you mean that image retrieval is an image classification? Image classification is a method to find labels. But image retrieval is not. $\endgroup$
    – David
    May 13, 2022 at 16:58
  • $\begingroup$ We can label image pairs similar based on belonging to the same class. It is not ideal, but it does make sense to me: many models are continuous and slightly larger dissimilarity should be reflected in slightly smaller score assigned by a model. $\endgroup$
    – Valentas
    May 13, 2022 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.