1
$\begingroup$

I have created a neural network for sentiment analysis using bidirectional LSTM layers and pre-trained GloVe embeddings.

During the training I noticed that the nn.Embedding layers with the freezed embedding weights uses the whole vocabulary of GloVe:

(output of the instantiated model object) (embedding): Embedding(400000, 50, padding_idx=0)

Also the structure of the nn.Embedding layer:
self.embedding = nn.Embedding.from_pretrained(embedding_matrix, freeze=True, padding_idx=self.padding_idx)

, where embedding_matrix = glove_vectors.vectors object and glove_vectors = torchtext.vocab.GloVe(name='6B', dim=50)

400,000 is the shape of glove_vectors object (meaning 400,000 pre-trained words in total).

Then I noticed that the training of the LSTM neural network took approximately 3 to 5 minutes per epoch. Which is quite too long for only 150,000 trainable parameters. And I was wondering if this had to do with the use of the whole embedding matrix with 400,000 words or it’s normal because of the bidirectional LSTM method.

Is it worth to create a minimized version of the GloVe embeddings matrix from the words that only exist in my sentences or using the whole GloVe embeddings matrix it does not affect the training performance?

$\endgroup$

1 Answer 1

1
$\begingroup$

Long Short Term Memory (LSTM) can take a long time to train because of the complexity of the architecture.

If you think the size of the embedding space is also slowing down training, you can reduce the size of the vocabulary by only taking the most frequently occurring words. Since it appears you are using PyTorch, you can do this within the torchtext.vocab.GloVe class with the max_vectors argument which limits the size of the loaded set.

$\endgroup$
1
  • $\begingroup$ Hey Brian thanks for the reply. So basically, you say that the size of the GloVe vocab/vectors (400,000) doesn't really affect the performance but rather the NN complexity does due to the use of LSTM layers. I will also try the trick with the max_vectors argument to reduce the size of loaded vectors from 400,000 to 100,000 vectors. $\endgroup$
    – NikSp
    Jan 12, 2023 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.