2
$\begingroup$

I started dabbling in neural networks quite recently and encountered a situation which is quite strange (at least with my limited knowledge).

The problem I'm using a NN is a regression problem which tried to predict the sales of a product for a particular kind of promotion in an FMCG.

Although the data is not strictly time series, it still has some time related attributes like a prediction may depend on a similar kind of promotion last year (which I've modelled using feature engineering).

Now to the problem I'm having:

I took the data from 2015-2017,augumented it by adding small amount of noise, shuffled it, and ran it through a neural network(I think the architecture is not important, but let me know if it is, I'll try and post it).

  • Optimizer: Adam with a decay
  • Loss Function : Volume weighted mape
  • Validation set : Randomly selected 20% of the data

The network trained well and gave me errors as small as 8%. And both training and validation set errors decreased together and did not indicate an overfit.

But the kicker is that when I applied the algorithm to new unseen data(2018, first half), the errors rose to 55%. .

I tried researching the issue on Google but didn't find anything useful. What is happening here? Am I doing something wrong?

$\endgroup$
  • 1
    $\begingroup$ Hi there! I hope you have normalised your data exactly in a similar way when using NN's, Also do share your Architecture as well, and iirc there's exactly a past Kaggle Playground Competition almost identical to what your are doing! Have a look $\endgroup$ – Aditya Dec 26 '18 at 3:46
  • $\begingroup$ @Aditya - I normalized the data in one go, split 2018 and kept it aside, and then fed the 2015-2017 to the network which takes 20% validation split... The architecture, to put it simply, is embedding + 3 conv layers + 2 dense layers .. And yes... will check the kaggle playground out.. $\endgroup$ – Manu Joseph Dec 26 '18 at 6:40
  • $\begingroup$ Try to normalize the validation data similar to the test data. What do you observe? $\endgroup$ – Martin Thoma Dec 27 '18 at 6:59
  • $\begingroup$ as explained, validation and test data was normalized the same way. The issue was a data leak. I've put that up as an answer. $\endgroup$ – Manu Joseph Dec 27 '18 at 23:44
1
$\begingroup$

Did you split your „original“ data into Train/Dev/Test? if not, try splitting your data into three parts. Train the model on your Train set, tune your hyperparameters on your Dev set and test your network on the Test set. The validation error doesn’t give an unbiased error, because the hyperparameters are tuned to it. That’s why you should always test your model on a given Test set. If the error is also low for the Test set I would guess you have another problem:

Without knowing too much about your approach and assuming you have no bugs, my guess is that the high error is because your Validation and Test set come from different distributions. I assume that maybe sales in 2015-2017 are quite different to those in 2018.

You should also take a look at Andrew Ng's explanation of this particular problem: https://www.youtube.com/watch?v=M3qpIzy4MQk

$\endgroup$
  • $\begingroup$ Yes, I did split the data into three parts. In that terminology, 2018 is my test set, 2015-2017 is split into train and validation... anyways I'll also try splitting 2015-2017 into three and check the performance.. . $\endgroup$ – Manu Joseph Dec 26 '18 at 6:43
  • $\begingroup$ I just checked by using 2015-mid-2017 as the train-validation and rest of the year 2017 as blind test and got similar results.. huge gap between the validation and test error $\endgroup$ – Manu Joseph Dec 26 '18 at 6:57
  • $\begingroup$ Again, you shouldn’t manually split your data like so: say validation: start to mid 2017 and test: mid to end 2017. That is no good practice. You should rather choose random samples. But at this point I think, you have some kind of bug. It doesn’t make sense that the same data you validate and test on, give that different results. Do you validate on the whole validation set or only on batches of the val set? Try following: Test your accuracy after training, again on the validation set. If the error is high you definitely have some kind of bug. $\endgroup$ – oezguensi Dec 26 '18 at 13:41
1
$\begingroup$

I finally figured out the problem. I'm just posting it as the answer so that if somebody has the same problem, they can get a clue in the right direction..

It was an issue of data leak. I was augumented the data, but I augumented it before the train validation split and hence a lot of similar cases were present in validation and train sets which pushed up the performance.

When I augumented only the train set and gave the validation set specifically to the method, the strangeness disappeared.

Now the validation accuracy and test accuracy are comparable.

$\endgroup$
1
$\begingroup$

I am glad to see that you've solved your problem. So, just adding to all the great answers here,such problems may also arise when your validation techniques fail you. Sometimes what happens is the unseen data(test data; not part of your train set) and your train data have a lot of differences in feature distributions. In this case, your validation set score isn't a good indicator of how your model is going to perform in case of unseen data because your validation set is part of your train set. If the degree of similarity between feature distributions among the data is high, then your normal validation techniques work. In case they are not, I generally employ Adversarial Validation.
It helps pick those training examples as validation which are most similar to test set.

Here are the links for the same by fastml :

http://fastml.com/adversarial-validation-part-one/

http://fastml.com/adversarial-validation-part-two/

Hope this helps :)

$\endgroup$
  • $\begingroup$ +1 for the less known validation technique.... Can you also include a link to the description to be more complete? May be the github link github.com/zygmuntz/adversarial-validation or the fastml link $\endgroup$ – Manu Joseph Dec 27 '18 at 23:43
  • $\begingroup$ Thanks for the upvote :) Added the fastml links. $\endgroup$ – Mohit Banerjee Dec 28 '18 at 4:36
0
$\begingroup$

It's called the overfitting problem. You should consider to use some regularization methods such as dropout, weight decay, and data augmentation. In my opinion, we usually see this problem in case of the financial datasets.

$\endgroup$
  • $\begingroup$ If it is vanilla overfitting, shouldn't there be a gap between the train and validation set? But they are pretty close, but when I'm applying the model to unseen data, a.k.a test set, that is when I have a problem. And Regularization is applied in the form of BatchNormalization, and Dropouts. Data Augumentation is also done by adding a noise to the original data set. Not really sure what you meant by weight decay, but if it is the learning rate decay, yes, it is also applied. $\endgroup$ – Manu Joseph Dec 26 '18 at 11:24
  • $\begingroup$ He did explain, that his validation error is going done. If he did everything correct while training it can’t be to overfitting. But you might be right if his implementation of checking val sccuracy is wrong, overfitting seems like a good candidate. $\endgroup$ – oezguensi Dec 26 '18 at 13:47
  • $\begingroup$ weight decay is another name of L2 regularization method where we penalize the case of large weight values. $\endgroup$ – Nga Dao Dec 26 '18 at 13:57
  • 1
    $\begingroup$ As I understand, the training and validation sets are in the (2015-2017) period. However, the test set is from 2018. There may exist some companies whose stock price is very stable during the (2015-2017) period but varies a lot in 2018. $\endgroup$ – Nga Dao Dec 26 '18 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.