How to evaluate multi label image retrieval model

Question

I'm using a deep hashing model to search most similar images in a database (most similar to the image given as a query). I'm doing this on the coco dataset which has multiple labels per image. I'd like to evaluate the performance of the model but I'm not sure what type of metric should be used here.

If it was just a single label per image, I'd go for something like mean average precision (given a query image of a dog, check how many dog images the system retrieved, evaluate MAP). But this obviously cannot be used on the multi label task (given a query image of three classes, system retrieved image with just one of them, it's not totally correct but it's not incorrect either). So are there any commonly used metrics to evaluate this kind of tasks? If so, please refer me to them. Or do I have to come up with something own (maybe some kind of a weighted MAP)?

Answer 1

• First consider analyzing the results per class (with normal accuracy/F-score/whatever) - this way you can have a good idea of where the algorithm is doing well and where it may be improved.
• Consider only the exact match. It's a "hit" if the algorithm got the exact labels that it should have. You can then use the hit count to calculate precision, recall and F-score.
• Hamming Loss: this is a favorite of researchers in my experience, it represents the how many of the total labels were misclassified

The relevant Wikipedia article seems to agree with me and adds the Jaccard index to the mix.

The paper that presents the famous Classifier Chains method (READ J. et al, Classifier Chains for Multi-label Classification, 2009) uses four different evaluation methods: a accuracy variation that closely resembles Jaccard distance, a similarly changed F1-score, and a log-loss function. The fourth method they use to evaluate is the area under the Precision-Recall curve, but that's one I'd argue should not be used (see the work on Precision-Recall-Gain curves by Peter Flach).