# How to initialize word-embeddings for Out of Vocabulary Word?

I am trying to use CoNLL-2003 NER (English) Dataset and I am trying to utilize pretrained embeddings for it. I am using SENNA pretrained embeddings. Now I have around 20k words in my vocabulary and out of this I have embedding available for only 9.5k words.

My current approach is to initialize an array of 20k X embedding_size with zeros and initialize the 9.5k words whose embeddings is known to me and make all the embeddings learn-able.

My question is what is the best way to do this? Any reference to such research will be very helpful ?

I have also asked a similar question later on Stackoverflow, here is the link.

• Subword embeddings to the rescue! Welcome to the site!
– Emre
Jan 23, 2018 at 7:36
• Down-voted, as you have asked it also here, without providing the link or sharing the info. stackoverflow.com/questions/48395570/… Jan 23, 2018 at 16:26
• @geompalik I agree that this is a problem but I'm not sure this is the best solution, as the linked question actually has a later timestamp than this one. Jan 23, 2018 at 16:58
• @geompalik I have added the link to question on stackoverflow in latest edits. My apologies. Jan 24, 2018 at 4:59
• @Emre I have gone through the entire paper, this approach seems most suitable to my task. Jan 24, 2018 at 5:08

## 1 Answer

You have a few options here. Of these, I think 1 will be the easiest to implement, as it's a standard language model with an alignment term added to the loss. I'd recommend 2a if you think you have the time, as I imagine its performance might be much better.

# 1 Use existing corpus to learn embeddings for new words

You can do this as you describe, though I would initialize the unknown embeddings with random noise instead of zeros. This will probably do fine, but it suffers from the problem that the model has no way of knowing that the SENNA embeddings are to be trusted more than the randomly initialized ones. So, it will generally take more (time, examples, etc.) to train this one well.

Another option is to try to capture the difference between known embeddings and randomly initialized ones. My suggestion would be to create your own embeddings [20k x emb_dim] and initialize it however you want. Then add a penalty to your model's loss for deviation from the SENNA embeddings. This is what's done in bilingual embedding models like this one. This will push the known embeddings close to SENNA and allow the unknown ones to freely vary. You could also reduce the coefficient for this alignment penalty as training progresses.

You could either learn these embeddings on a dedicated embedding task (like CBOW) or as part of your primary task.

# 2 Hybrid model for unknown embeddings

A second approach is to forgo attempting to learn embeddings for your unknown words at all. Typically, mosty word RNN models do this for low-frequency words through something like the <unk> tag. For you, that would be half your vocabulary. Fortunately, there are a lot of tools for filling in these out-of-vocabulary (OOV) words.

## 2a Subword embeddings

As @Emre pointed out, subword embeddings have been used to great effect to solve out-of-vocabulary problems. Here's an example of a model that uses subword embeddings to achieve an "open vocabulary" model. This means you would combine your pre-trained embeddings (which you would not need to make learnable) with a character-level model to learn when and how to produce OOV words.

## 2b <unk> replacement

Another approach is to, as a post-processing step, replace the <unk> symbols with words from your corpus. You can build your own language model (even a simple markov chain would be an improvement). Also, some modern models have more sophisticated methods for replacement that work quite well.