2
$\begingroup$

I have 400 time series of length 50. 200 of them have values between 1-10 and are considered of type A. The rest 200 have values 1-10 with the exception that 3 from the total of 50 data points have value 20, and are considered of type B.

I am trying to use a RNN network to make it learn these differences using tensorflow.

I use GRU cell with input size is 50 (the whole time series), internal state is 100, 3 layers, feature length is 1 and dropout 0.8. Batch size is 1 (1 whole time series of 50 values) 80% of the series are used for training and 20% for test, evenly distributed between type A and type B.

I use softmax as activation function and for gradient descent optimization i use RMSPropOptimizer. Also i am not changing the data at all before feeding it into the neural network, i tried normalizing them using (x-min()/max()-min()) but it didn't work.

When i am trying to train it, i always get 50% error. Maximum epochs i tried are 10 and it still was 50%.

The same network can classify correctly if i use it on data that have way more differences than just on these 3 points, even on small dataset of about 50 runs of each type.

Is it impossible for a RNN to learn these small differences on just those 3 data points out of the 50? What would you suggest i can change in my configuration? I also tried with 2000 time series and still the error is the same.

Also note that these 3 data points exist at the beginning of the time series. (in case this has something to do with my issue).

$\endgroup$
2
  • $\begingroup$ So, you are classifying two categories: one with values 1-10 and one with mostly 1-10 but some 20? Why do you need an RNN for this task? Assuming you actually do need one, 3 layers may be too complex, try fewer. Try multiple dropouts and more epochs. Plot your train vs validation loss to get an insight into performance. $\endgroup$
    – Hobbes
    Nov 27, 2017 at 16:26
  • $\begingroup$ The reason i am trying to use RNN is because these two types might also have other differences not only this specific one. So its not so simple as to say "if you have value 20 then your are type B". I do use this more simple dataset to mostly test if RNN can find even those small differences. Also i tried with 2 layers and i get same issue. $\endgroup$
    – Ploo
    Nov 27, 2017 at 16:47

1 Answer 1

2
$\begingroup$

I let it run for more epochs, as suggested, and i noticed that after 23 epochs, rnn did get 0% error.

I then tried changing activation function of GRUCell from the default tanh to relu and it greatly speed up training. It took 1-4 epochs to get 0% error.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.