3
$\begingroup$

I am analyzing a time-series dataset using (supervised) tensorflow deep learning. The tensorflow code is given a series of inputs, and based on each input, the NN has to predict output value in near future.

For training there are lots of pre-labeled input/output of past. Each input is an array of state per time step, and each input is labeled with a result value. So far this kind of task is common at deep learning, and I used multiple GRU layers to solve this.

But problem is, I found that the pattern for finding appropriate output actually changes as time goes. For every 10 hours (approximately), the pattern changes so at every start of each pattern period (like first 1-2hours of total 10 hours) the NN is bad at predicting, but after that, prediction rate enhances. Plus, I assume the data also has some noise too.

So far my implementation with GRUs do its job, but I want to find better way to build NN if possible. I currently know some basic supervised learning techniques and some advanced like LSTM, and for unsupervised learning I know DQN and policy-gradient.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Could this be a sampling problem as opposed to an architecture problem? Ie do you sample the inputs during training for all time periods? Another architecture to add to LSTM is a gated neural network, which has a master network with N gates which direct inputs to N LSTMS, so if you decide that there are 4 patterns, the master network would predict which of those 4 that input goes to, and then the slave LSTM does the actual predicting work. More on idea here (this is an atypical use of it fyi) https://blog.openai.com/learning-a-hierarchy/. $\endgroup$
    – Pavel Savine
    Apr 28, 2018 at 16:39
  • $\begingroup$ @Xylitoly How do you know the pattern changes over time? Is that just a gut feel? $\endgroup$
    – horaceT
    Apr 28, 2018 at 20:07
  • $\begingroup$ @horaceT No, the pattern is actually made by an heuristic function made by human. In this case it is widely known that pattern actually changes over time. The heuristic function can be a variety of things, such as stock market graph or playstyle of a sports player. $\endgroup$
    – Xylitoly
    Apr 29, 2018 at 11:59

2 Answers 2

Reset to default
3
$\begingroup$

Do you know when does the new pattern start? You could reset the hidden state of the RNN each time this happens during training. A more detailed explanation in this paper: https://arxiv.org/pdf/1706.04148.pdf

In that paper, they try to predict the next item in a user's session. They use a GRU to create a user-level representation (which in your case could be the pattern representation) and another GRU to predict the next item based on the current session information.

To do that, they reset the hidden state of the second GRU every time the session changes and reset the hidden state of the first GRU every time the user changes, and I think you could reset the hidden state every time the pattern changes. It is difficult to explain it all right here, I recommend you to read the paper I mentioned before.

EDIT:

To know where are the pattern change points you could train another network, maybe using as labels points where your GRU's accuracy decreases substantially, and then use this network's predictions to reset the hidden state on these points at training time as I mentioned before.

Improve this answer
$\endgroup$
4
  • $\begingroup$ This sounds like a classic regime change question. Why not have the RNN learn the change points? $\endgroup$
    – horaceT
    Apr 28, 2018 at 20:06
  • $\begingroup$ Just because it doesn't learn the change points, that's why this question is being asked. You could use a different network to learn them, as stated by @Pavel Savine in the comments, and it is completely possible to use that information to reset the hidden state in the approach I posted as an answer. $\endgroup$
    – Andres Espinosa
    Apr 28, 2018 at 22:23
  • $\begingroup$ @Andres Espinosa - To piggy back off the changing hidden state idea since I have never used it: If one needed to keep some amount of information previous 'phase state', is it conceivable to 'dilute' the hidden state instead of complete re-initialization by adding some amount of noise or near-zeros (like a dropout layer) - perhaps the amount of noise/near-zeros is determined by the output magnitude of whichever second neural net you choose for marking that cutoff point - or does that reduce to something simpler by the math...? $\endgroup$
    – Pavel Savine
    Apr 29, 2018 at 0:20
  • $\begingroup$ @Pavel Savine - What you say makes sense since the cutoff point is stochastic, but it's just difficult to tell before trying it. Interesting idea though. $\endgroup$
    – Andres Espinosa
    Apr 29, 2018 at 4:38
1
$\begingroup$

Going off the discussion in the @Andres Espinosa's answer, here is an option that might work for this without any extra gating voodoo. The Nested LSTMs paper, Implementation here. The nested LSTM idea is to build in the idea of a temporal abstraction hierarchy - which in your case could possible capture the current pattern at the first level, and the need to switch between patterns at a deeper level.

In addition there is the untouched idea about why your LSTM is not working in the first place. Before doing extra work, maybe it is worth trying visualizing the LSTM activations. In theory, if the LSTM has learned different phases in your input, there would be a different distribution of activations depending on the 'phase' or more significant changes in the hidden state between phases vs within the phase, since it would pay attention to different things depending on the phase, it could be a matter of just more training time. If it is consistently not picking up something that can be visually diagnosed, then it would suggest trying one of the other ideas.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.