We are doing our Thesis on multimodal retrieval. it's basically searching different modalities (multimedia ex: text, video, images ...) with other modalities. i.e. searching a database of images with a text query.
For any modality we need first to map it to a space where it has a constant number of features, and those feature must be somehow expressive of the data.
For images, papers we evaluate utilizes SIFT feature extraction, we use NUSWIDE for evaluating different methods, it already exists in SIFT format so most papers we evaluate uses these existent dataset.
We tried to improve on this feature extraction mechanism by using Inception or Resnet and taking layers exactly before softmax as our features. however, they perform way worse than SIFT. we used tensorflow and keras to extract features.
so any idea why would resnet/inception perfrom worse than SIFT?