2
$\begingroup$

I have a neural network to solve a time series forecasting problem. It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features. The performance of the model is average and I would like to investigate whether adding or removing features will improve the performance. I have constructed the neural network using keras.

The features I have included are:

  1. The historical data
  2. quarterly lagged series of the historical data (4 series)
  3. A series of the change in value each week
  4. Four time invariant features tiled to extend the length of the series. (another 4 series)

I am aware I could run the model many times changing the combination of features included each time. However, along with tuning the hyperparameters (for it might be that 8 features works really well with one set of hyperparameters but not with another set) this is really a lot of possible combinations.

Is there any separate way that I can use to guage if a feature is likely to add value to the model or not?

I am particuarly concerned that I have four time-invariant features being fed into the model which is designed to work with time varying data and I would like a way to measure their impact and if they add anything or not?

$\endgroup$
  • 1
    $\begingroup$ You basically want to assess the statistical significance of your features. There's no native inexpensive way to do with with a neural network. However, you could do this by fitting a separate regression with ARMA errors model and check coefficient p-values. $\endgroup$ – Digio Jan 27 '19 at 14:20
  • $\begingroup$ What you suggest sounds interesting, but could you expand upon your idea a little bit, I didn't fully understand. $\endgroup$ – Aesir Jan 27 '19 at 14:27
  • $\begingroup$ You could try fitting a type of linear model to your series, using your neural network features as the dependent variables, then look at coefficient p-values to see which features have important impact to the series. There are many ways to do this, R has regression with ARMA errors (package forecast), python has the GLSAR class, and with some extra manual work you can do this using simple linear regression. $\endgroup$ – Digio Jan 27 '19 at 15:26
5
$\begingroup$

Don't remove a feature to find out its importance, but instead randomize or shuffle it.

Run the training 10 times, randomize a different feature column each time and then compare the performance. There is no need to tune hyper-parameters when done this way.

Here's the theory behind my suggestion: feature importance

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.