In a neural network architecture can I use the sigmoid function in some layers and the tanh function in the others? Is it a good choice?


Yes you can. There are no hard rules against having different activation functions in any layer, and combining these two types should give no numerical difficulties.

In fact it can be a good choice to have tanh in hidden layers and sigmoid on the last layer, if your goal is to predict membership of a single class or non-exclusive multiple class probabilities. The sigmoid output lends itself well to predicting an independent probability (using e.g. a logloss (aka cross-entropy) objective function).

Whether or not it is better than using sigmoid on all layers will depend on other features of your network, the data and the problem you are trying solve. Usually the best way to find out which is better - at least in terms of accuracy - is to try out some variations and see which scores best on a cross-validation data set. In my experience, there often is a small difference between using tanh or sigmoid in the hidden layers.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.