0
$\begingroup$

I've developed a deep learning model trained from scratch on fruits and vegetables. However, as the data is limited, I can only cover a few different types of fruits and vegetables with the model alone. However, the accuracy on those categories is very high, 90% training and validation accuracy with around 80% testing accuracy (from a different set of data).

Now I tried to use this trained model on unseen data. So I've removed the softmax layer and only kept the other layers of the model. The idea is to use the feature vector and cosine similarity to compare vegetables.

Let's say bananas were in my original dataset but apples and oranges were not. I use the model to extract features of apples and oranges off of google, then compare the vectors of apples with origins. I would expect the cosine similarity to be very low as they are very different vegetables. However, I get a cosine similarity of 0.9, which is very confusing. To test my sanity, I used a pretrained Mobilenet v2 model and also removed the last layer. Then used the same method to extract the features of apples and oranges. Now they only have a similarity of around 0.5, which is much more reasonable.

I have a few ideas as why this could be the case, perhaps the model could not generalize past the categories given to it during training, or it did not learn similar features that could be translated to apples and oranges so it could not differentiate between the two. However I have no certain explanation. Does anyone have any idea why this is happening, and/or how I can prevent this from happening?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
0
$\begingroup$

The differentiation of shape starts in the initial layers. Colour information is exploited in further layers. Your network has less layers, try increasing your layers (keep filters low in the initial layers).

Also, if you are looking at similarity models, look at contrastive training or ArcFace loss or siamese type models. If you want to stick with classification networks, look at GeM pooling.

Best way would be to use MobileNet (or better networks such as EffNetB4) to finetune with GeM pooling.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the information, very helpful. I still do not understand why I was able to achieve such high accuracy on the training set though? It seems counterintuitive that it is able to perform well on the training set, while bombing on the unseen data. The shapes of the fruits and vegetables do vary a lot so it would make sense, even for a model with few layers, to perform well. $\endgroup$
    – ZWang
    Commented Apr 21, 2021 at 13:04
  • $\begingroup$ You are correct. $\endgroup$
    – Abhishek Verma
    Commented Apr 21, 2021 at 13:50
  • $\begingroup$ datascience.stackexchange.com/questions/94071/… Hi, could you take a look here? $\endgroup$
    – x89
    Commented May 6, 2021 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.