0
$\begingroup$

I use a Recurrent Neural Network for time series forecasting of electrical load data from a cooling device based on past values of the load time series and temperature values. I first normalize the data before the training. So the time series has very low values after the normalization (between 0 and 1). Because of this the training error (root mean squared error) for the training and evaluation set is quite low (smaller than 0,05). However when using the test data, the root mean squared error is extremely high (13754.9). I use 2 previous days as input and as output is one day ahead (with a time resolution of 15 minutes).

Now I basically have 2 questions:

  1. As I normalize the data before training, the general error is consequently quite small during training. Can this be a disandvantage for the optimizer "adam" because I can imagine that so small values might somehow "mislead" the optimizer.
  2. Do you have any suggestion how I can improve the results? So I have a extremely low error on the training and validation set (due to the small values) but an extremely high error on the test set.

Here you can see the error plot of the training for the training and validation data set: Loss function during training

Here you can see the actual and forecasted values of the test data set: Forecasted and actual values on the test data set

Reminder: Does nobody have any suggestions? Are my questions unclear or do you need further information? I'd appreciate every comment and will be quite thankful for your help.

$\endgroup$

1 Answer 1

1
$\begingroup$

It is hard to answer your question(s) as stated without knowing more about how you are splitting your data. How did you split the data? Are the distributions of the values of your features similar amongst your splits?

EDIT What I meant by "distribution of values" is probably best illustrated by plotting some of your features as a histogram for the training, validating, and testing set. From there you can spot check a specific features histogram between the splits to see if say, for example, the training set has a more gaussian look while the other splits are very skewed. This difference in distribution will affect the performance of your model.

For more information on this, you can read the following article which also links to other helpful article on the subject.

https://www.section.io/engineering-education/correcting-data-shift/

$\endgroup$
6
  • $\begingroup$ Thanks Miguel for your answer. Basically I use a 70/20/10 split. Meaning I have 70 % of the data for training, 20 % for validating and 10 % for testing. However, I do not understand what you mean by saying "Are the distributions of the values of your features similar amongst your splits?" $\endgroup$
    – PeterBe
    Aug 30, 2021 at 7:28
  • $\begingroup$ @PeterBe I have edited my answer to address your concern. $\endgroup$ Aug 31, 2021 at 21:18
  • $\begingroup$ Thanks Miguel for your answer and effort. Basically in my data I have a shift between training and testdata. However, this is because of the temperature. As I am predicting cooling loads it is clear that in summer time the loads are higher whereas in winter there is no cooling load at all. But when I exclude the winter data for training, the results get even worse. I read the link that you posted. While it was definitely informative, the methods to tackle the problem of shifts in data mentioned there are not at all applicable in my example (and I reckon they can't generally be appplied often) $\endgroup$
    – PeterBe
    Sep 2, 2021 at 14:50
  • $\begingroup$ Can you share this data with me in some way so that I can look at it deeper? If this is not possible, when the data is split is there a balance between the different seasons in testing, and training data? $\endgroup$ Sep 2, 2021 at 20:02
  • $\begingroup$ Thanks Miguel for your answer and effort. I really appreciate it. Could you send me maybe an E-Mail adress such that I can try to share a link with you for the data (I do not necessarily want to post a link to the data here publicly such that everyone could download it) $\endgroup$
    – PeterBe
    Sep 3, 2021 at 7:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.