0
$\begingroup$

In my NLP task, I use Glove to get each word embedding, Glove gives 50 float numbers as an embedding for every word in my sentence, my corpus is large, and the resulted model is also large to fit my machine, I'm looking for a way to reduce each word embedding from 50 numbers to something less, maybe one floating number is possible, is there an approach to do that?

I was thinking about an aggregation approach,like taking the average of the 50 numbers, would that work?

$\endgroup$
3
  • 1
    $\begingroup$ Can you try to make this question a bit more specific, and provide further context? $\endgroup$
    – Warlax56
    Feb 14 at 15:14
  • $\begingroup$ I edited my question and tried to make it more clear. $\endgroup$
    – sakher
    Feb 14 at 15:46
  • $\begingroup$ hi @sakher feel free to approve any answers that was useful or follow up if you’d like more explanations. Cheers $\endgroup$
    – eliangius
    Feb 17 at 19:34

2 Answers 2

0
$\begingroup$

Taking the average (or any other statistic combo) of the embedding vector for each word is NOT a good idea because the embedding dimensions are independent & you will loose a lot of nuance of the learned space.

You could do a few things however. First would be to do PCA on the embedding matrix to reduce variance a little bit say to 45/50 dimensions. This is a quick hack so don’t over do it. Ideally you would train another embedding of of this one into a smaller dimensionality space say of 10-30 dimensions that gives you good results for your task. You could remove unused or very rare words from your embedding altogether. You could also normalize your embedding vocabulary to exclude case sensitive and accented words.

From an engineering standpoint if fitting in RAM is still an issue you could also load chunks of the embedding vocabulary into memory at a time via some caching mechanism like LRU.

$\endgroup$
0
$\begingroup$

You can use the binarization method of image embeddings in image retrieval tasks like https://arxiv.org/pdf/1609.08291.pdf

However, binary inputs might be tricky as neural network input. Initialization plays an important role and pure ones and zeros might hinder the model from converging. So, you might need to apply layer normalization or some other technique to stabilize the training.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.