In Efficient Backprop (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), Lecun and others propose to use activation function that don't reach target values on their asypmptotes.

They explain (§ 4.5) why an activation function that only reach target values asymptotically may result in drawbacks : instability and ouptu saturation.

On of their proposed solution is to (carefully) use activation function that have a range greater than [-1,1]. For that purpose they propose to add a linear term to the standard tanh activation function, or add a multiplicative constant >1.

This make some sense, however, I am wondering how to use that ouptut. I am used to rescale [0,1] scores to calibrate models in probability. But I don't know what I would do with a 1.2 score. Perform as usual with a monotonous function to transform score in probabilities ? Add another layer ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.