Inception/ResNet doing worse than SIFT in feature extraction

Question

We are doing our Thesis on multimodal retrieval. it's basically searching different modalities (multimedia ex: text, video, images ...) with other modalities. i.e. searching a database of images with a text query.

For any modality we need first to map it to a space where it has a constant number of features, and those feature must be somehow expressive of the data.

For images, papers we evaluate utilizes SIFT feature extraction, we use NUSWIDE for evaluating different methods, it already exists in SIFT format so most papers we evaluate uses these existent dataset.

We tried to improve on this feature extraction mechanism by using Inception or Resnet and taking layers exactly before softmax as our features. however, they perform way worse than SIFT. we used tensorflow and keras to extract features.

so any idea why would resnet/inception perfrom worse than SIFT?

Answer 1

After examining the dataset, we found that the problem was in NUSWIDE dataset itself. Almost half of the dataset didn't have supervised labels (81 labels entered by humans). Also we weren't training on the whole dataset, rather, we were taking random samples because some methods required that the whole dataset was loaded in memory which wasn't feasible. After knowing that, we deleted the tuples that didn't have supervised labels, extracted features and done PCA and got a comparable results to SIFT (improved, in some cases). So my advice if you face a similar problem is:

• Make sure that your data labels are valid.
• Use multiple layers for feature extraction, not just the deeper ones in the network.
• Using PCA instead of random sampling dataset worked very well for us.