0
$\begingroup$

Usually in a LSTM network, we have certain parameters that need to be set before the model can begin training. I am specifically talking about vocabulary size, padding length and embedding dimension. Below is a simple LSTM network where I have randomly chosen the 3 parameters:

vocab_size = 20000
pad_size = 35
embedding_dim = 50

# ONE HOT REPRESENTATION FOR TRAINING SET
ohr_train = [one_hot(i, vocab_size) for i in train_x]

# PAD EACH TEXT IN TRAIN SET SO THAT EACH TEXT IS OF SAME LENGTH
train_embedded_docs = pad_sequences(ohr_train, padding = 'pre', maxlen = pad_size)

# DEFINE THE MODEL
model = Sequential()
model.add(Embedding(vocab_size, dimension, input_length = pad_size))
model.add(Dense(1, activation = 'sigmoid'))

My question is how do you set all the three above mentioned parameters?

PS: From the answers I understood how to set vocabulary size. Padding length should be more than the maximum length of text in train set (not the whole set as that would lead to data leakage!).

But when setting the embedding dimension via HP tuning, it will be a time consuming process as for each combination, I would have to run the whole model and as you know neural nets take a long time to run. Isn't there a better way?

$\endgroup$

2 Answers 2

1
$\begingroup$

Vocabulary size, padding length and embedding dimension are like hyperparameters which needs to chosen wisely to get good performance from model

Vocabulary Size : The set of unique words used in the text corpus is referred to as the vocabulary. When processing raw text for NLP, everything is done around the vocabulary. When the text corpus is large and you need to limit the vocabulary size to increase training speed or prevent overfitting on infrequent words. To do this most people restrict it to specific number or say apply a threshold for example Vocabulary size is equal to words which have frequency greater than 10.

Padding Length : Since LSTM takes input of same length, inputs are padded to the maximum length of the sequence in the batch

Embedding Diemnsions : Usually people use multiple of 2 like 128, 256 and 512. Higher the dimension better the capturing of information but more the time required for training

$\endgroup$
1
  • $\begingroup$ Kindly see the updates answer. $\endgroup$
    – spectre
    Jan 27 at 11:48
0
$\begingroup$

Ideally padding should not be done for whole data but for batches. Padding comes from the need to encode sequence data into contiguous batches: in order to make all sequences in a batch fit a given standard length, it is necessary to pad or truncate some sequences.

Usually people do post padding i.e. introduce zero after the actual text but pre padding is also an option

But as padding does not have actual data, model should be informed at some part of the data is actually padding and should be ignored. That mechanism is masking.

$\endgroup$
7
  • $\begingroup$ Batches as in the batch_size we pass during model.fit()? If so what difference does it make if I pad the whole train set vs pad each and every batch? Also any articles/tutorials on masking? I heard it for the first time. $\endgroup$
    – spectre
    Jan 27 at 12:03
  • $\begingroup$ Please use this. tensorflow.org/guide/keras/masking_and_padding $\endgroup$ Jan 27 at 12:04
  • $\begingroup$ The difference is in one batch you may have maximum length to be 50 only while in other batch it can be 40..It saves a lot of padding effort and training time $\endgroup$ Jan 27 at 12:05
  • $\begingroup$ Ok got it. Now for embedding dimension, is HP tuning the only option? $\endgroup$
    – spectre
    Jan 27 at 12:13
  • $\begingroup$ I would suggest reading a few architecture which have worked for similar problem. Also its always that you have enough data more dimension in embedding give better results. HP should be the last choice $\endgroup$ Jan 27 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.