I am building a neural network that consists of an LSTM, dense and dropout layers using Keras to forecast 8 continuous values in the future. I unfortunately have a very small data-set of 56 observations. I divided my data-set into 48 observations for training and 8 observations for testing. I ran my code 5 times in order to split my data-set differently each time. Within each run, I also fit the model 100 times and use the maximum test R-squared to pick the best performing model in each run. Therefore, I have the following r-squared values: 0.89, 0.90, 0.93, 0.91, 0.87. If I want to report an r-squared value, which one should I use? How to be sure that I reached the global optimum r-squared value? Also, would the RMSE benefit me in taking such decision?

  • $\begingroup$ $R^2$ is a flawed metric for a nonlinear model like a neural network. The total sum of squares does not decompose into the sum of squares of the regression and the sum of squares of the residuals. There is a third term. In the case of linear regression, that term winds up being zero. In nonlinear regression, the term is not zero. $\endgroup$ – Dave May 11 at 0:28
  • $\begingroup$ @Dave then what metric should I use when reporting the accuracy? I know that I can use a metric like RMSE to assess the performance of the model but it might be easier for communication if there is a percentage indicating the performance. $\endgroup$ – Karim Afifi May 11 at 15:47
  • $\begingroup$ I'm still relatively early in the machine learning journey, however, I would think you would measure based on your loss/accuracy in the model's history. r = model.fit( ... ) r.history['loss'] r.history['accuracy'] $\endgroup$ – Sean Payne May 14 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.