3
$\begingroup$

I am trying to predict TimeSeriesA by using a CNN. I create snapshot images of the timeseries and these are then labelled.

With a very simple snapshot I get reasonable training and test accuracy. When I apply the model to real world in production I also get reasonable accuracy.

Inorder to improve the accuracy, I added other timeseries to the snapshots that may or may not add value.

Both my training and testing accuracy increased (Training much more so). However, my production accuracy has gone down greatly.

Why might this happen? The original data is still in the snapshot in exactly the same format. Can a CNN be confused (wrong word!) by the additional data?

Here is a look at the stationarity of TimeSeriesA:

Year: 2000 Ave: -0.0003 Std: 0.0076 Skew: 0.2166
Year: 2001 Ave: -0.0002 Std: 0.0072 Skew: 0.0158
Year: 2002 Ave: 0.0006 Std: 0.0056 Skew: -0.2445
Year: 2003 Ave: 0.0007 Std: 0.0065 Skew: -0.0402
Year: 2004 Ave: 0.0003 Std: 0.0067 Skew: -0.2640
Year: 2005 Ave: -0.0005 Std: 0.0056 Skew: 0.2420
Year: 2006 Ave: 0.0004 Std: 0.0047 Skew: 0.2711
Year: 2007 Ave: 0.0004 Std: 0.0039 Skew: -0.3177
Year: 2008 Ave: -0.0001 Std: 0.0087 Skew: 0.3768
Year: 2009 Ave: 0.0001 Std: 0.0076 Skew: 0.2327
Year: 2010 Ave: -0.0002 Std: 0.0074 Skew: 0.0112
Year: 2011 Ave: -0.0001 Std: 0.0074 Skew: -0.2599
Year: 2012 Ave: 0.0001 Std: 0.0051 Skew: 0.2541
Year: 2013 Ave: 0.0002 Std: 0.0046 Skew: 0.0818
Year: 2014 Ave: -0.0005 Std: 0.0039 Skew: -0.1489
Year: 2015 Ave: -0.0004 Std: 0.0076 Skew: 0.2973
Year: 2016 Ave: -0.0001 Std: 0.0051 Skew: 0.0076
Year: 2017 Ave: 0.0005 Std: 0.0045 Skew: 0.3101
Year: 2018 Ave: -0.0002 Std: 0.0045 Skew: -0.1658
Year: 2019 Ave: -0.0002 Std: 0.0033 Skew: -0.1124

I train the model with data up to end of 2010. I get a training accuracy and validation accuracy of around 65%.

When I then appy the model to data from 2011 to 2019, I get a drop-off in accuracy. The green years are those included in the training.

enter image description here

Improve this question
$\endgroup$
1
  • $\begingroup$ It looks like your production data is significantly different from the data you used for testing, that's what you should investigate imho. $\endgroup$
    – Erwan
    Commented Jun 13, 2019 at 14:44

1 Answer 1

Reset to default
3
+25
$\begingroup$

Obviously, without knowing the exact data, everything I will say is mere speculations. Here are some of my guesses why this might happen:

  • Your production data has a different distribution from your training data

This can happen, data analysis would be required to double check how your production data might deviate. If this is the case, I would investigate what might cause this, is it something you can adapt to? Or is it due to an external factor you can measure?

  • Your network is too large and overfitted artifacts in training

If your network is large, you might have overfitted on certain artifacts that are present in your train and test set and therefore, it performs badly on your production data. I would train smaller networks or on many smaller snapshots to double check that your model can learn appropriately.

  • Your evaluation method is not appropriate

When training time series, make sure to always evaluate on future data and not past (that's cheating!)

  • Bugs

Obviously, bugs may always be the cause of some issues. Maybe preprocessing is doing something funky.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Re your first point. I have added the return stats for each year in my data. Does that suggest the distribution is changing? $\endgroup$
    – ManInMoon
    Commented Jun 17, 2019 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.