I have an LSTM network and am testing it on some dummy ARMA signals. I'm trying to predict the signal 5 time steps into the future.
The network is capable of outperforming Naive (persistence) when predicting the signal. My next toy example was to make a multi-variate prediction where I take several ARMA signals (ARMA_1 and ARMA_2) and the target variable is a linear combination of these signals (0.5 * ARMA_1 + 0.5 * ARMA_2) 5 time steps into the future. The training dataset is just the original ARMA signals and the target is just the linear combination with the offset.
However, while the training loss improves, the validation loss gets worse pretty much as soon as the predictions deviate away from flat. So the model is overfitting.
The network is already pretty simple (8 hidden nodes in the LSTM and a couple of hidden layers with 16 nodes in them). The networks won't over-fitting in the univariate case so I don't think it should be an issue overfitting in the multivariate case. Any simpler a network and it isn't capable of modelling the univariate case, so I don't want to simplify it further.
There is more than enough samples in the dataset that it should be able to learn the signals. I've done runs with dropout, regularisation and different learning rates to no avail.
I was wondering if anyone had any advice for why the networks is overfitting here but not in the univariate case, even with the same network parameters?