How to set vocabulary size, padding length and embedding dimension in LSTM network?

Question

Usually in a LSTM network, we have certain parameters that need to be set before the model can begin training. I am specifically talking about vocabulary size, padding length and embedding dimension. Below is a simple LSTM network where I have randomly chosen the 3 parameters:

vocab_size = 20000
pad_size = 35
embedding_dim = 50

# ONE HOT REPRESENTATION FOR TRAINING SET
ohr_train = [one_hot(i, vocab_size) for i in train_x]

# PAD EACH TEXT IN TRAIN SET SO THAT EACH TEXT IS OF SAME LENGTH
train_embedded_docs = pad_sequences(ohr_train, padding = 'pre', maxlen = pad_size)

# DEFINE THE MODEL
model = Sequential()
model.add(Embedding(vocab_size, dimension, input_length = pad_size))
model.add(Dense(1, activation = 'sigmoid'))

My question is how do you set all the three above mentioned parameters?

PS: From the answers I understood how to set vocabulary size. Padding length should be more than the maximum length of text in train set (not the whole set as that would lead to data leakage!).

But when setting the embedding dimension via HP tuning, it will be a time consuming process as for each combination, I would have to run the whole model and as you know neural nets take a long time to run. Isn't there a better way?

Answer 1

Vocabulary size, padding length and embedding dimension are like hyperparameters which needs to chosen wisely to get good performance from model

Vocabulary Size : The set of unique words used in the text corpus is referred to as the vocabulary. When processing raw text for NLP, everything is done around the vocabulary. When the text corpus is large and you need to limit the vocabulary size to increase training speed or prevent overfitting on infrequent words. To do this most people restrict it to specific number or say apply a threshold for example Vocabulary size is equal to words which have frequency greater than 10.

Padding Length : Since LSTM takes input of same length, inputs are padded to the maximum length of the sequence in the batch

Embedding Diemnsions : Usually people use multiple of 2 like 128, 256 and 512. Higher the dimension better the capturing of information but more the time required for training

Answer 2

Ideally padding should not be done for whole data but for batches. Padding comes from the need to encode sequence data into contiguous batches: in order to make all sequences in a batch fit a given standard length, it is necessary to pad or truncate some sequences.

Usually people do post padding i.e. introduce zero after the actual text but pre padding is also an option

But as padding does not have actual data, model should be informed at some part of the data is actually padding and should be ignored. That mechanism is masking.