conv net data retrieval on unseen class

Question

I have build a conv net for image classification which work "well"

Now I extract features from last fully connected layer and use it for image retrieval (find image most similar to my target image) using hamming distance. it's working prety well even if I'm not able to predict how it'll be rotation invariant, sensible to noise..

I have try image retrieval only on class that have been seen by my model while training (but not training data). Do you know if it could work on class that have never been seen by the model while training?

e.g

Let's says model has been train on car and truck. I want to find most similar image to that yellow small car and it's work well it's return me yellow and small car.

Now let's imagine I apply it on a new data set with Plane. I want to find most similar image to that small yellow plane. Should I expect to find yellow small plane or that'll be totaly random?

Since network has never seen plane while training, is it possible to predict, or at least have an intuition, of the result?

Answer 1

It depends. Let me explain you what features are detected when you train a classification model.

Let's take same example as yours - car and truck classification. The CNN will extract features from both type of images - tires, windows, shape, texture etc.

Then it'll separate the features - if it sees something like bigger tires, bigger windows, rectangular shape, it'll give more weight to truck class. Similarly for car class.

Now coming to your image retrieval part. This is what you're doing - passing all the image through CNN and extracting features for each image. Then comparing your test image features with other features to find most similar one.

Now think, what if you pass new image to the same CNN ? What features will be extracted.

CNN will extract the same features as it used to for car and truck - tires, windows, shape etc. Because it's trained for that only. It won't start extracting wings as feature if you pass airplane.

So if you pass airplane as test image - same features as car and truck would be extracted and you will get results based on the comparison of these features. Hence if you pass yellow small airplane, you might get yellow small car having same tire size as your airplane or yellow small truck having shape similar to your plane or in best case yellow small airplane.