1
$\begingroup$

I have a transformer network that is trained on time series data. The task is to predict if a variable will increase a certain percentage in the next 7 days. The input is data from the 90 previous days. The training data is data from 2000 up till 2023, while validation data is from 2023 and until todays date. There is no overlapping data in the training and validation set. I train the model on the training data and make note of the epoch number $n_E$ where the validation loss is minimized. Then I concatenate the training and validation data and retrain for $n_E$ epochs to get the final model. However, when concatenating the data I have more batches in the resulting training set and I am concerned for overfitting and that there is a lot of variance in the epoch number $n_E$. Despite this, I think that I will loose some relevant new information if I do not retrain on the concatenated data set, or if I leave out a final test set. I think cross validation will not be feasible due to the time it takes to train the model. Any suggestion of a robust way of performing the training?

Appreciate any help that can make my model perform when in production

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Unfortunately, time series needs some data preprocessing in order to be efficient. Furthermore, some NN are more noise-sensitive than others. Others are more adapted to medium-term prediction or short-term prediction.

My best advice is to understand your model's limits & behavior with docs or articles on the Internet.

You can also detect the cycles: it is not necessarily 90 days. You can find it doing some genetic algorithm, using the time cycle as a variable.

Then, you can compare 2 or 3 different models (e.g., Prophet, Mamba, LSTM, etc.). Meta's Prophet is the easiest one. Finally, if your signal has noise, reducing it with a smoothing algorithm could improve results.

There is no easy solution: Trying many settings and methods is the best guarantee to get great results.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for your reply. So what is your suggestion for a training strategy? For example training on the first 80% of the time series, early stopping on the next 10% validation data. Finally testing on the last 10%?. An alternative I have thought of is concatenating the training and validation data after early stopping, and training for an additional epoch with a smaller learning rate. Before finally testing on the test data. $\endgroup$
    – QCQCQC
    Commented Jan 10 at 12:44
  • $\begingroup$ Yes, it is a good start. Nevertheless, I highly recommend evaluating your model predictions over time = checking the quality of the predictions every day with the actual values. Very often, there is a bias due to model adjustment over time. $\endgroup$
    – Nicolas Martin
    Commented Jan 10 at 14:10
  • $\begingroup$ Thank you for your help. I have put together a script that evaluates my model in production in real time. $\endgroup$
    – QCQCQC
    Commented Jan 11 at 10:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.