# Are RNN or LSTM appropriate Neural Networks approaches for multivariate time-series regression?

Dear Data Science community,

For a small project, I've started working on Neural networks as a regression tool, but I am still confused about possibilities of some variants. Here's what I am aiming to do:

• I have multiple input data time series $$X(t)=[X_1(t), X_2(t), X_3(t),X_4(t)]$$, and multiple target data time series which I want to modelize $$Y(t)=[Y_1(t), Y_2(t)]$$. All data are available for training.
• I aim to train my model/regression on an interval $$[t_0,t_n]$$, and then be able to apply it on a larger different interval.
• I know that relation between my $$Y$$ and $$X$$ are non-linear, but also that in need to take in account lag, or inertia. For example, $$Y_1(t)$$ is dependant of $$X_1([t-dt_1,t])$$ and $$X_2([t-dt_2,t])$$. All $$dt_n$$ are different, but I have an approximate idea of how 'far' I need to reach.

With this in mind, through some research I have been guided to focus on Recurrent Neural Networks (RNN) and Long Short Term Memory (LSTM) networks. I aim to use TensorFlow/Keras to work on this. However, after some reading, I'm getting confused with those solutions. Many people present them in prediction applications, which supposedely means that those networks use data from a time interval (either $$X$$ or $$Y$$ in my case) on an interval $$[t_0,t_n]$$ to predict $$Y(t_{n+1})$$. But my objective is to use $$X([t_0,t_n])$$ to modelize $$Y([t_0,t_n])$$ (on the same time interval). I am getting confused with this notion of "prediction".

Therefore, are RNN and LSTM networks appropriate solutions for my multivariate time series regression/model project? Or am I already going the wrong way?

As a beginner in this field, any reference or link to ressources/tutorial, or demo, is also gladly welcome.

Here is a really good source to begin multivariate time-series forecasting in Keras using LSTMs.

I aim to train my model/regression on an interval $$[t_0,t_n]$$ and then be able to apply it on a larger different interval.

There's no harm in this as long as you perform the right kind of multi-step forecasting. If your problem requires you to train on $$[t_0, t_n], \text{for some } n < 100$$ and produce $$y_1(t_{n+1}), y_2(t_{n+1})$$ as outputs then APPLY it on $$[t_n, t_n+100]$$ then there will be issues in implementation, as most ML models will require you to provide the same shape as you did when you were training your model. Here's when sliding across time-series will help you. Simply put,

\begin{align} \text{Train on: }& [t_0, t_n] &\text{ Output: }& y_1(t_{n+1}), y_2(t_{n+1}) \\ \text{Predict on: }& [t_1, t_{n+1}] &\text{ Output: }& y_1(t_{n+2}), y_2(t_{n+2})\\ \text{Predict on: }& [t_2, t_{n+2}] &\text{ Output: }& y_1(t_{n+3}), y_2(t_{n+3})\\ & \vdots & & \vdots\\ \end{align}

I know that relation between my Y and X are non-linear, but also that in need to take into account lag or inertia. For example, Y1(t) is dependent of X1([t−dt1,t]) and X2([t−dt2,t]). All dtn are different, but I have an approximate idea of how 'far' I need to reach.

Well, you can model your time-series data as $$X_1(t-k),\cdots,X_1(t), X_2(t-d),\cdots,X_2(t)$$ as $$X$$ (input) and $$X_1(t+1), X_2(t+1)$$ as $$y$$ (target), where $$k$$ and $$d$$ refer to different lags for each variable.

• Thank you for your answer. For me, it now makes a lot more sense for this prediction/forecasting approach, by sliding. If I'm not mistaken, I have data from t0 to t3000, and that I train my algorithm on [t0,t100], by applying on succesive intervals [tn,tn+100], I'll be able to generate Y from t101 to t3001 (although t3001 state does not interest me). I'll looking at this tutorial which indeed seems a good intro for a Keras/TF application. Jan 14 '19 at 11:12
• Exactly! Glad it helps. I would appreciate it if you upvote and accept the answer! Jan 14 '19 at 11:21

RNN are appropriate for modeling time series. Their recursive input provides possibliity to model dynamic systems basing of their time series. I would recommend create two different models for each output. I am personally using MATLABs NARX networks for making predictions models for insdustrial purposes and they do a thing. https://www.mathworks.com/help/deeplearning/modeling-and-prediction-with-narx-and-time-delay-networks.html;jsessionid=bb13c3b99f070979e406ab13ad33 here is a ref for how to use NARX networks for time-series prediction.
I do not know about doing it in TF/Keras but RNN should have good performance.

• Thank you for your anwser. Yes, making two models seems the best way to obtain best accuracy in my scenario. Regarding NARX approach, however, is that it formulates Y(t) as f(X(t-1), ..., X(t-d), Y(t-1), ..., Y(t-d)) while I'm trying to formulate Y(t) as f( X(t), ..., X(t-d) ). Matlab proposes the Nonlinear Input Output model, which ressembles what I am aiming to do. However, it kinda lack freedom on the network's customization. Maybe I am missing something available in the matlab toolboxes ? Jan 14 '19 at 10:19
• I am using Neural Networks Toolbox from MATLAB 2017b. Using a GUI doesnt give a freedom in network customization by doing it by scripts/functions do a thing for me. You can pick layers' count, count of neurons in each layers, transfer functions, input signals, training algorithm and so on. mathworks.com/help/deeplearning/ref/feedforwardnet.html Here is a doc with examples how to train feedforward network which might be enough if your model doesn't need to model dynamic properties. Jan 14 '19 at 10:39
• mathworks.com/matlabcentral/answers/…. Here is an example how to change net's transfer function. Number of layers and neurons could be manipulated by hiddenSize argument in feedforwardnet(hiddenSize, trainAlgorithm). [10 7 5] for example, means network with 3 hidden layers with 10, 7 and 5 neuron each respective layers. You mentioned you need to take inertia in account. Using that feedback Y(t-d) input could help you with that. I am using it for identification of dynamic systems and it works Jan 14 '19 at 10:42
• Oh my... I'm just realizing that I never knew you could input an array in HiddenSize in order to add multiple layers. Well thank you for that, I'll have a lot of thing to try with matlab. I'll also have to check how to change the activation function and/or to use gated cells. Jan 14 '19 at 11:00