Pattern such as [1 0 0 0], [0 1 0 0], [0 0 1 0], [0 0 0 1] can easily be learned by using LSTM. We have created patterns where above mentioned vectors serve as the basis but we reveal one index randomly and if the random index contains 1 we let the vector as it is and if random index contains zero then we change the entry to -1. For example, [1 0 0 0] remains the same if randomly selected index was 0. However, if index was any other value the vector looks like [0 -1 0 0](index 1 was chosen). Based on this we created a new data set, LSTM can learn and predict these patterns as well but takes more time and examples before converging. However, if the vector length goes above 6 i.e. [0 0 0 0 0 0 -1] then LSTM is unable to correctly predict all the indices. This is probably because there are more zeros in the vectors and merely one non-zero entry, therefore, it is considered as noise. I have tried using lower learning rate, two LSTM layers, larger batch-size but the network is unable to reach above 90% accuracy. Is there anyway to tune the neural network so that it starts learning these sparse vectors or a way we can skip these zeros and focus on non-zero entries?