Initial embeddings for unknown, padding?

Question

Last time I've been passing pretrained word embeddings into LSTM to solve text classification problems. Usually, there are additional <pad>, <unk> replacements for padding and unknown types. Of course, there are no pretrained vectors for them.

Solutions I've come up with are

1. Fill them with random values
2. Fill them with zeros

Which approach is better/wrong? What is the common practice?

Note: I use pytorch+torchtext if it matters.

Answer 1

In my experience, what works well is :

• For padding, fill a zero vector embedding (as pixel intensity in image data padding) is the only and best solution.
• For words that don't have a pre-trained embedding, you should try to train them: as you do, fill them with random values when initializing, but set them to trainable.

Answer 2

That will vary depending on how and what framewoirk are you using, for example using tensorflow with a tf.nn.dynamic_rnn, if you pass the length sizes:

https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn

sequence_length: (optional) An int32/int64 vector sized [batch_size]. Used to copy-through state and zero-out outputs when past a batch element's sequence length. So it's more for correctness than performance.

Hence it will not train over those sequences.

However if you need to pass the padded tokens, it will depend mostly on how you manage your LSTM initial state. If you have a zero initial state and want to maintain it that way you will pad with zeroes, equally if you have a random initial state you will pad with random numbers.

Unless you senunknownd the sequence_length in general you will need to use different tokens for padding and unknown words.

Here are some words on pading: (Note that when it uses tf.nn.dynamic_rnn it feeds the sequence_length):

https://r2rt.com/recurrent-neural-networks-in-tensorflow-iii-variable-length-sequences.html

This is an analysis of different ways of manage the LSTM initial state (Note that when it uses tf.nn.dynamic_rnn it does not feed the sequence_length):

https://r2rt.com/non-zero-initial-states-for-recurrent-neural-networks.html

Lastly, you need to train the unknown token. One thing that could improve your embedding is to categorize your unknown tokens for example with regex (If they are like verbs, adjectives, nouns, numbers, etc) . The end of the following article under the title "Unknown Words" gives some ideas.

https://stathwang.github.io/part-of-speech-tagging-with-trigram-hidden-markov-models-and-the-viterbi-algorithm.html