Is this a problem for a Seq2Seq model?

Question

I'm struggling to find a tutorial/example which covers using an seq2seq model for sequential inputs other then text/translation.

I have a multivariate dataset with n number of input variables each composed of sequences, and a single output sequence which is unrelated to any of the input variables (e.g. using weekly wind speed and humidity to predict temperature).

I've converted the features and label to batches with fixed time steps and n dimensions, however I'm confused which model should be used. Ideally the output should be a single sequence (e.g. annual temperature) however which model would achieve this? Is this something an LSTM could achieve, or is this a problem for a seq2seq model?

Any suggestions/insightful would be appreciated.

Answer 1

To use a seq2seq neural network for timeseries regression (and not forecasting, as everybody seems to be doing), a simple Keras model could be

inputs = Input(shape=(n_timesteps, n_features))
x = LSTM(n_units, return_sequences = True)(inputs)
outputs = Dense(1, activation="linear")(x)

where you output a single value in the final layer, with linear activation, so that no scaling is applied. Then a common choice is to minimize MSE with your favorite optimizer - usually RMSprop for timeseries.

Then, if you add convolutions and pooling layers, be sure to set padding = "same", to maintain the sequence length throughout the layers, or fix downsampled timesteps with RepeatVector to match the input sequence length. Specific choice of layers may be problem-dependent.

Answer 2

The type of your problem is time series regression. The common way of dealing with such problems is using ARMA models and Kalman filters with input (control vector). Look at this links:

https://www.sas.upenn.edu/~fdiebold/Teaching104/Ch14_slides.pdf

https://en.m.wikipedia.org/wiki/Kalman_filter

For time series regression using LSTM, check Multivariate LSTM Forecast Model part from this link:

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras