1
$\begingroup$

Imagine you have trained a model containing an Embedding layer.

Your model performs well and you're happy with your embedding.

Then, suddenly, you want to add a new item in your vocabulary. In other words you want to compute the embedding of this new item.

Basically an Embedding layer is a lookup table, used to turn positive integers into dense vectors of fixed size, and now you want to consider a new integer that was not there during the training.

How can you do this without retraining the model from scratch?

Does it make sense (after some matrix shape adjustement) to relaunch the training freezing all the parameters except the ones used to embed the new item?

$\endgroup$

1 Answer 1

2
$\begingroup$

If outright retraining is out of the question (which honestly makes sense since it'd likely be time and resource intensive), then you can try a couple things:

  1. Treat this as an out-of-vocabulary problem and keep an OOV token that you apply to unseen words
  2. Partial retraining as you described: there are methods (I haven't used them so YMMV) where you add new rows for your unseen words, fix the parameters corresponding to your existing vocabulary, and you train and update the new rows. Example PyTorch discussion
  3. Imputing OOV: FastText and other methods that learn character/sub-word level embeddings could be useful for assembling new words.LOVE FastText
  4. Transfer Learning where you get a larger pre-trained embedding layer that hopefully has the vocabulary you need

EDIT: This also could be connected to this question here

$\endgroup$
1
  • $\begingroup$ I'm trying 2) but it's not trivial.... I'll update here a solution if I find one :) Unfortunately the other options are not applicable to me (I'm not embedding words but a categorical feature) $\endgroup$
    – MarcoM
    Feb 16 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.