0
$\begingroup$

In a deep learning network (CNN or RNN), we might use word embeddings such as FastText, Glove, etc. to represent the input text. My question is:

If I'm working on a data from Twitter, and I have a variety of text lengths, as in the picture below (example):

enter image description here

sometime I have a few sentences of a length larger than 150 and the average length of the rest of the sentences is 48.

Here, what I noticed in some implementations online that they pad the short sentences with "PAD" word to increase their size to reach the length of the largest sentence (Red spaces in the picture), where this PAD word is filled with a random value. Is it a good practice to do so?

If my dataset contains 5000 tweets, and I have about 20 tweets with a length larger than 150 , the random value will affect the classification task with a high ratio since most of the sentences will be padded with a random value.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Yes, when you have different sequences with different length you can pad all the sequences so that they have all the length of the longest one. is a good practice and doesn t impact the learning mode. Another solution could be online learnig. With online learning you give to the model 1 sample at the time (batch_size =1) and in keras you can train the model with batch with different lengths.

With online learning your computation is slower, but theoretically your model need less epochs to learn.

If you want to pad you should pad with all 0s and not random values.

Improve this answer
$\endgroup$
2
  • $\begingroup$ thanx, but is there any reference for: online learning ? $\endgroup$
    – Minions
    Nov 4, 2018 at 11:48
  • $\begingroup$ here there is an example: datascience.stackexchange.com/questions/26366/…, edit: that example is most associate with only different length for different batches is not online learning, anyway if you take mini_batch_size = 1 you have online learning. You can also train as in the example, with batch_size > 1 and with different batches using different lengths, but if you have as in your example only a single sequence with th longest length, then you are forced to use online learning. $\endgroup$
    – erre4
    Nov 4, 2018 at 12:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.