Say I have access to several pre-trained CNNs (e.g. AlexNet, VGG, GoogleLeNet, ResNet, DenseNet, etc.) which I can use to extract features from an image by saving the activations of some hidden layer in each CNN. Likewise, I can also extract features using conventional hand-crafted techniques, such as: HOG, SIFT, LBP, LTP, Local Phase Quantization, Rotation Invariant Co-occurrence Local Binary Patterns, etc. Thus, I can obtain a very high-dimensional feature vector of an image that concatenates the individual features vectors outputted by these individual algorithms. Given these features, and given a data set of images over which I want to perform similar image retrieval (i.e. finding the top-k most similar images to a query image X), what would be the most appropriate ways to implement this task?

One possible idea I have in mind is to learn an image similarity embedding in euclidean space by training a neural network that would receive as input the aforementioned feature vectors, and perhaps down-sampled versions of the image as well, and output a lower dimensional embedding vector that ideally should place similar images close to each other and dissimilar images far apart. And I could train this network using for example Siamese Loss or Triplet Loss. The challenge of this approach though is generating the labels for the (supervised) training itself. For example, in the case of the Triplet Loss I would need to sample triplets (Q,X,Y) and somehow determine which one between X and Y is most similar to Q, in order to generate the label for the triplet (i.e., in order to "teach" the network I need to know the answers myself beforehand, but how? I guess this is domain dependent, but think of challenging cases where you have very heterogeneous images, such as photography galleries, artwork galleries, etc).

Anyways, this is just an idea and by no means I pretend to mean this is the right approach. I'm open to new suggestions and insights about how to solve this task.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.