I am building an LSTM with keras which have an activation parameter in the layer. I have read that scaling on the output data should match the activation function's output values.

Ex: tanh activation outputs values between -1 and 1, therefore the output training (and testing) data should be scaled to values between -1 and 1. So if the activation function is asigmoid the output data should be scaled to values between 0 and 1.

Does this hold for all activation functions? If I use ReLu as activation in my layers what should the output data be rescaled to?


What you read hold true for the neurons of the output layers and not for the hidden layers!

Hence, its true that if you are using tanh in output layers then you need the data labels to be within [-1, 1] where as between [0, 1] for sigmoid.

As for your concern with relu, use it on output layers if you know that the range of the labels of the data is positive only. If you are using relu in hidden layers then the scaling doesn't depend on relu but rather on the type of activation function used in output layers.

  • $\begingroup$ I know for sure that the range of the labels is positive. so what should be the scaling of these labels if I know this information? $\endgroup$ Apr 27 '20 at 16:18
  • $\begingroup$ Is there any natural upper bound on the data labels? $\endgroup$ Apr 27 '20 at 20:54
  • $\begingroup$ there is not no. $\endgroup$ Apr 28 '20 at 14:07
  • $\begingroup$ In such a case relu would be suitable. Using sigmoid would restrict the output to 1 which is not good for your case. $\endgroup$ Apr 28 '20 at 20:06
  • $\begingroup$ Do you still have some doubt? $\endgroup$ May 1 '20 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.