I want to extract features from pre trained
ResNet model for over 2M data. Problem? Even with the average pooling applied on the last layer's result, it provides a feature vector of length
2048 which looks something like
[0.2,0.4,0.5,0.01,0.003,0.09....], if I convert to
200K images, I need
1.5GB of memory. But I have a bigger problem at hand now with a very huge dataset. I need to have a good algorithm which can decrease the dimensionality so that I can use any of
ANNOY, LSH, SCANN, HNSWLIB etc for recommendation.
there are a few techniques that know but not sure about which one do good in case of Image Embeddings. Can someone suggest me which one would be good for my use case?