Designing a pretrained DNN for image similarity

Question

I am pretty new to deep learning and really hope that you can help me.

I want to write a python program that lets me choose an area in a reference image. This subimage of variable size should then be used to search in a database of images. Then the parts of the images with the highest similarity to the reference sub image should be given. However I have big problems with the sizing of the reference and database images. I tried to read into pretrained DNNs (like VGG19) and use the features of a last layer for similarity computation. But these DNNs seem to accept input arrays only in certain resolutions. Should I then rescale the reference image? The database images will most likely be much larger than the reference. Should I then partition all the database images into smaller subsets? Or use a single shot algorithm like YOLO?

Since there are so many different algorithms I would be very thankful for every comment or idea.

Lasse

Answer 1

That is not currently possible.

There is no such thing as a database that supports arbitrary content-based image indexing or image queries. Image Databases (IDBs) are indexed by R-tree family or members of the quadtree family. Those only support limited SQL (text) queries.

You would have search image-by-image for each query image which would defeat the purpose of having a database.

Answer 2

A net like VGG19 has a part that generates a representation of an image, followed by classification layers. You can use the part that creates the representation and run that on all your subimages, as well as all images in your DB that you want to compare to. Then you can compare the representations to find the most similar image, for example using nearest-neighbor (just find the example that has the smallest distance in the representation space, but there are faster ways to do that than just calculate all distances every time).

Nets like VGG19 require a fixed image size/resolution. You can use specialized nets that do not have this requirement, but you can try that later. For now, just rescale your images, or crop them, or both (you can crop in multiple ways to get multiple representations that all point back to the same original image).

Final step. Rather than using nearest neighbors to find the most similar representation, you can train the classification layers to do this for you. I’m not sure this is going to help your results but I think it’s worth a try.