Pattern such as [1 0 0 0], [0 1 0 0], [0 0 1 0], [0 0 0 1] can easily be learned by using LSTM. We have created patterns where above mentioned vectors serve as the basis but we reveal one index randomly and if the random index contains 1 we let the vector as it is and if random index contains zero then we change the entry to -1. For example, [1 0 0 0] remains the same if randomly selected index was 0. However, if index was any other value the vector looks like [0 -1 0 0](index 1 was chosen). Based on this we created a new data set, LSTM can learn and predict these patterns as well but takes more time and examples before converging. However, if the vector length goes above 6 i.e. [0 0 0 0 0 0 -1] then LSTM is unable to correctly predict all the indices. This is probably because there are more zeros in the vectors and merely one non-zero entry, therefore, it is considered as noise. I have tried using lower learning rate, two LSTM layers, larger batch-size but the network is unable to reach above 90% accuracy. Is there anyway to tune the neural network so that it starts learning these sparse vectors or a way we can skip these zeros and focus on non-zero entries?

  • 1
    $\begingroup$ why dont you just use the index of the non-zero value instead of the whole sequence, eg for [0,0,1,0,0,0] <=> 2 (index 2 is non-zero rest are zero). This is a one-to-one mapping and avoids the sparse vector problem $\endgroup$ – Nikos M. May 11 at 13:38
  • $\begingroup$ How can I use indices for training? Any reference or article that can help in this regard? $\endgroup$ – Python_Learner May 11 at 13:50
  • 1
    $\begingroup$ You use the index as value instead of vector $\endgroup$ – Nikos M. May 11 at 13:51
  • $\begingroup$ yeah you are right but this limits me to only a single non-zero entry case. As I progress further I may have two or more non-zero entries with larger vector sizes. $\endgroup$ – Python_Learner May 11 at 14:06
  • 1
    $\begingroup$ then use a vector of indices instead of very sparse vectors $\endgroup$ – Nikos M. May 11 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.