We are doing our Thesis on multimodal retrieval. it's basically searching different modalities (multimedia ex: text, video, images ...) with other modalities. i.e. searching a database of images with a text query.

For any modality we need first to map it to a space where it has a constant number of features, and those feature must be somehow expressive of the data.

For images, papers we evaluate utilizes SIFT feature extraction, we use NUSWIDE for evaluating different methods, it already exists in SIFT format so most papers we evaluate uses these existent dataset.

We tried to improve on this feature extraction mechanism by using Inception or Resnet and taking layers exactly before softmax as our features. however, they perform way worse than SIFT. we used tensorflow and keras to extract features.

so any idea why would resnet/inception perfrom worse than SIFT?

  • $\begingroup$ Did you train the networks first or did you use pre-trained? $\endgroup$ – geometrikal Jul 3 '17 at 22:06
  • $\begingroup$ a pretrained version of course! $\endgroup$ – mohRamadan Jul 4 '17 at 13:08
  • $\begingroup$ Which features did you take from CNNs? Higher layers detect geometric information/edges and deeper more semantic information $\endgroup$ – Alex Aug 5 '17 at 1:41
  • $\begingroup$ We used what's explained in the tutorials mentioned above, we used the features from the layer right before the softmax for both inception and resnet. $\endgroup$ – mohRamadan Aug 6 '17 at 13:41

After examining the dataset, we found that the problem was in NUSWIDE dataset itself. Almost half of the dataset didn't have supervised labels (81 labels entered by humans). Also we weren't training on the whole dataset, rather, we were taking random samples because some methods required that the whole dataset was loaded in memory which wasn't feasible. After knowing that, we deleted the tuples that didn't have supervised labels, extracted features and done PCA and got a comparable results to SIFT (improved, in some cases). So my advice if you face a similar problem is:

  • Make sure that your data labels are valid.
  • Use multiple layers for feature extraction, not just the deeper ones in the network.
  • Using PCA instead of random sampling dataset worked very well for us.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.