So I have around 600 classes and 17k sentences for text classification.The feature vectors are trained using word2vec.
Vector_size = 100 ; LSTM hidden units= 100
I am using class weight for imbalance problem.I wanted to know if LSTM is affected by sparsity in training data when max length is kept 18 but minimum length of sentences is 1.
And, how can we use keras train_on_batch to create batches of 5,10,25 for variable length instead of padding.
The solutions for sequences with different length are:
1- padding, but you do not want to.
2-truncate the sequences, so if you have for instance 3 sequences of different lengths you truncate the sequences so that they have all the length of the shortest one.
3- using timesteps of 1, but in this case your model will not learn the dependencies between different timesteps
4- using online learning with batch size = 1.
For the fourth solution this is an example in keras to create a random time-length batches of training data: Training an RNN with examples of different lengths in Keras. So you can have different batches with different lengths, and since you are using online learning with batch_size = 1 this can be a good solution. The fourth solution can also have advantages and disadvantages. For online learning the training is slower, but usually you need less epochs compared to sgd with batch size > 1 to have a good model. If you use stateful lstm, this could lead you to an unstable model, not always, but can happen.
