I want to extract features from pre trained ResNet
model for over 2M data. Problem? Even with the average pooling applied on the last layer's result, it provides a feature vector of length 2048
which looks something like [0.2,0.4,0.5,0.01,0.003,0.09....]
, if I convert to float32
for 200K
images, I need 1.5GB
of memory. But I have a bigger problem at hand now with a very huge dataset. I need to have a good algorithm which can decrease the dimensionality so that I can use any of ANNOY, LSH, SCANN, HNSWLIB
etc for recommendation.
there are a few techniques that know but not sure about which one do good in case of Image Embeddings. Can someone suggest me which one would be good for my use case?
Some of the algorithm defined are in this and this blog but none them tells which one is better for embeddings.
k
clusters out ofN
images. And then within the clusters, search the image similarity BUT!! again, to make clusters, I have to get the data at ONCE for training and then apply some clustering method on the vectors, which again creates the problem of dimensionality. $\endgroup$ – Deshwal Jan 16 at 18:13