# On the choice of LSTM input/output dimension for a spatio-temporal problem

I am using LSTM neural networks from (R)Keras for a matter of spatio-temporal interpolation. I manage to get the network to output predictions but the results are not outstanding (very little improvement on validation loss). I am wondering about the shapes of training data and labels.

Say I have 50 dates of measurements of the variable of interest $$y$$, accompanied by about 100 covariates $$x$$ (spatial coordinates, temperatures...). Each date has 24 measurements of $$y$$, so nsamples=50*24=1200. If I set the timestep hyperparameter of LSTM to e.g. 3, and use a moving window of step 1, I have therefore an input table $$X$$ of shape (1200, 3, 100).

On the other hand, should the labels table $$Y$$ be of dimension (1200, 3) or (1200, 1) ? More precisely, which of the following describes the problem the best: $$(X_{n,t-2} ; X_{n,t-1} ; X_{n,t}) \rightarrow (Y_{n,t-2} ; Y_{n,t-1} ; Y_{n,t})$$

$$(X_{n,t-2} ; X_{n,t-1} ; X_{n,t}) \rightarrow (Y_{n,t} ; Y_{n,t} ; Y_{n,t})$$

$$(X_{n,t-2} ; X_{n,t-1} ; X_{n,t}) \rightarrow Y_{n,t}$$

$$(X_{n,t-2} ; X_{n,t-1} ; X_{n,t}) \rightarrow Y_{n,t+1}$$

Or are they all plausible ways of addressing slightly different problems? As I said, I'm trying to spatially interpolate $$Y$$ for the 50 dates of measurements, as well as predicting $$Y$$ for the year(s) to come. So I expect one is more relevant than the others but I have no clue on which one.

I hope this is understandable as I clearly miss some technical vocabulary here.

On time-series models

All models that you have mentioned are correct and practical depending on the problem (the index $$n$$ is not required). The second one however produces redundant results which is a waste of computation. Even $$(X_{t} ; X_{t+1} ; X_{t+2}) \rightarrow (Y_{t-1})$$ is correct, if you are fitting on an archive and want to predict a year given the covariates from the next three years.

But only the last model $$(X_{t-2} ; X_{t-1} ; X_{t}) \rightarrow (Y_{t+1})$$ is a forecasting model. So in general, if you want to interpolate into the next $$k$$-th year from now $$t$$, you should use: $$(X_{t-2} ; X_{t-1} ; X_{t}) \rightarrow (Y_{t+k})$$

or

$$(X_{t-2} ; X_{t-1} ; X_{t}) \rightarrow (Y_{t+1},...,Y_{t+k})$$ Even a better model that takes advantage of known $$Y$$'s in the past would be:

$$(X_{t-2}|Y_{t-2} ; X_{t-1}|Y_{t-1} ; X_{t}|Y_{t}) \rightarrow (Y_{t+k})$$

where $$|$$ denotes vector concatenation to produce a 100 + 1 dimensional vector for each known year.

As a personal opinion, for the time-series prediction task, 24 data points per year is very small compared to the dimension of $$X$$, which is 100. 1200 samples for $$X \rightarrow Y$$ regression (ignoring the time) is more practical; if selecting 10 from 100 covariates is possible even better.

Because of the small data set, I would suggest:

1. $$(Y_{t-m} ;...; Y_{t-1} ; Y_{t}) \rightarrow (Y_{t+k})$$ for time series prediction, and

2. $$X \rightarrow Y$$ regression for estimating the relation between X and Y.

Relation to LSTM and RNN

If we use LSTM/RNN to model time-series, they would be stateful. That is, when input $$X_{t-2}$$ is fed to an LSTM, it keeps an internal state (hidden state) to be combined with the next input $$X_{t-1}$$ and so on. Regarding the input/output dimension, here is an RNN animation from a post on medium by Raimi Karim that shows an arbitrary step among 3 steps of feeding $$(X_{t-2} ; X_{t-1} ; X_{t})$$ to the network:

As you see, dimension and number of inputs are independent of output. We can feed 5 inputs $$X_{t-4}$$ to $$X_{t}$$, each 100 dimension (100d) and receive a 1d output by setting the dimension of hidden states to 1d, or setting it to 10d and use an extra dense layer at the end to convert 10d to 1d, or receive a 50d output, or a 150d (three 50d) output, etc.

Word "stateful" in Keras (source)

LSTM and RNN are stateful by definition, this [badly named] variable in Keras means

If stateful=True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch. Fabien Chollet

For example, if each batch has 24 samples indexed from 0 to 23 (each sample could have the form $$(X_{t-2}, X_{t-1}, X_{t}, Y_{t+1})$$), then the last hidden state $$h$$ from 8th sample will be used as the initial hidden state for 8th sample in the next batch. Except for special cases that there is a temporal order between batches and their samples, this must be set to False.

• Thank you, this is clear! So the choice for any of those problems would only depend on how I arrange inputs and outputs, right? So the network does not need any more argument to be trained for one problem or the other. If you mind answering that also: does any of this problem needs stateful LSTM? Or should preferentially be addresed with stateful LSTM? I am very confused by this argument, and nothing I have read so far makes it clear wether I should use those or not (and if so, how should I modify the formulas above). Mar 26, 2019 at 9:23
• Thanks for the edit and the reference! Just to be clear, currently I am not using stateful=TRUE. Does this mean that the network is not building links between $X_{t-1}$ and $X_{t}$? If so, what is the interest of feeding the network with this time-batch structure when stateful=FALSE? I get it's a different question, you don't have to answer. Thanks again! Mar 26, 2019 at 10:53
• Well thanks again for this last edit, you really helped me there! All the best Mar 26, 2019 at 11:36
• If you don't mind answering this last question: in the predictive case $(X_{t-2}|Y_{t-2} ; X_{t-1}|Y_{t-1} ; X_{t}|Y_{t}) \rightarrow Y_{t+1}$, how should I input $X_{t+1}$? I'm asking that because lots of my covariates are static and can therefore be useful in predicting the process in the years to come. Mar 26, 2019 at 15:45
• @YoB. if $t+1$ denotes "next year" we have no access to $X_{t+1}$ in real-time cases. Btw, since each timestamp should be 101 dimension, you can use $(...X_{t}|Y_{t};X_{t+1}|0) \rightarrow (Y_{t+1})$ just using a dummy 0, but I think it may cause under-performance, try it. You can also use the previous year: $(...X_{t}|Y_{t};X_{t+1}|Y_{t}) \rightarrow (Y_{t+1})$. Mar 26, 2019 at 15:55