Goal: trying to use walk-forward validation strategy with keras tuner for time series when training a neural network (mainly LSTM and/or CNN).

Did anyone find a direct way of doing this?

One possible way I can think of is:

  • implementing a custom 'objective' function, e.g. 'mean squared error' using walk-forward validations trategy
  • this custom function could be passed via the 'objective' input parameter in the tuner (be it RandomSearch, BayesianOptimization, Hyperband...)
  • when calling tuner.search, we pass the train & validation data, to be used in the walk-forward custom function

Thanks in advance

  • 1
    $\begingroup$ There is no question here. Can you clarify it, please? $\endgroup$ – juanba1984 Apr 30 '20 at 9:24
  • $\begingroup$ Already added the explicit question. Nevertheless, I think that is not a reason for giving a downvote on an interesting and necessary topic $\endgroup$ – German C M Apr 30 '20 at 9:55

I finally came out with a way to implement bayesian hyperparameter optimization for a time series neural network model using walk-forward validation. You can find a gist with the code here

I used some helper functions for the walk-forward validation from this Jason Brownlee book

It is basically as follows:

  • original time series data: enter image description here

  • define a function to convert your time series dataset into a supervised dataset format, another one for the train-test split (ordered in time as done in sklearn.model_selection.TimeSeriesSplit)

enter image description here

  • define your evaluation metric, root mean squared error in this case
  • we also add a differencing function, in case we want to make our time series stationary (by differencing, we can reduce trend and/or stationality)
  • define the walk-forward validation functions (walk_forward_validation and repeat_evaluate)
  • define the keras tuner bayesian optimizer, based on a build_model function wich contains the LSTM network in this case with the hidden layers units and the learning rate as optimizable hyperparameters

  • define the model_fit function which will be used in the walk-forward training and evaluation step

enter image description here

  • lastly, find the evaluation metric value and std enter image description here

To keep in mind: this is a proof-of-concept example about how to use keras tuner for time series using walk-forward validation. Further analysis could be done to improve forecasts.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.