I started dabbling in neural networks quite recently and encountered a situation which is quite strange (at least with my limited knowledge).
The problem I'm using a NN is a regression problem which tried to predict the sales of a product for a particular kind of promotion in an FMCG.
Although the data is not strictly time series, it still has some time related attributes like a prediction may depend on a similar kind of promotion last year (which I've modelled using feature engineering).
Now to the problem I'm having:
I took the data from 2015-2017,augumented it by adding small amount of noise, shuffled it, and ran it through a neural network(I think the architecture is not important, but let me know if it is, I'll try and post it).
- Optimizer: Adam with a decay
- Loss Function : Volume weighted mape
- Validation set : Randomly selected 20% of the data
The network trained well and gave me errors as small as 8%. And both training and validation set errors decreased together and did not indicate an overfit.
But the kicker is that when I applied the algorithm to new unseen data(2018, first half), the errors rose to 55%. .
I tried researching the issue on Google but didn't find anything useful. What is happening here? Am I doing something wrong?