In my NLP task, I use Glove to get each word embedding, Glove gives 50 float numbers as an embedding for every word in my sentence, my corpus is large, and the resulted model is also large to fit my machine, I'm looking for a way to reduce each word embedding from 50 numbers to something less, maybe one floating number is possible, is there an approach to do that?
I was thinking about an aggregation approach,like taking the average of the 50 numbers, would that work?
Taking the average (or any other statistic combo) of the embedding vector for each word is NOT a good idea because the embedding dimensions are independent & you will loose a lot of nuance of the learned space.
You could do a few things however. First would be to do PCA on the embedding matrix to reduce variance a little bit say to 45/50 dimensions. This is a quick hack so don’t over do it. Ideally you would train another embedding of of this one into a smaller dimensionality space say of 10-30 dimensions that gives you good results for your task. You could remove unused or very rare words from your embedding altogether. You could also normalize your embedding vocabulary to exclude case sensitive and accented words.
From an engineering standpoint if fitting in RAM is still an issue you could also load chunks of the embedding vocabulary into memory at a time via some caching mechanism like LRU.
You can use the binarization method of image embeddings in image retrieval tasks like https://arxiv.org/pdf/1609.08291.pdf
However, binary inputs might be tricky as neural network input. Initialization plays an important role and pure ones and zeros might hinder the model from converging. So, you might need to apply layer normalization or some other technique to stabilize the training.
