i have created some logistic regression model (different preprocessing) with softmax function. and i mix all model with an ensemble with a hierarchical method. so the output of all model (base) will be used as input for the final model (logistic regression too).

the default base model used a softmax function. i think transform a confident value into a probability will lose much information. so i have a plan to change the softmax into activation function.

what i learn in my CNN class, Relu is the best default activation function for an image. but my case is multi-class classification email.

which activation function should i choose? Relu, sigmoid or other?


  • $\begingroup$ LeakyRelu, Relu6 or rather you can create your own Relu version with adaptivity on the -ve axis $\endgroup$ – Aditya Oct 2 '18 at 5:55

I think you will have to experiment; there isn't generally a one-activation-fits-all for hierarchical models. Give the ones you mentioned a try, but perhaps starting with sigmoid and tanh and also a LeakyReLU.

The reason I'd perhaps leave out a normal ReLU at first is because that too would potential trim out information before the final model, as it simply removes negative weights to zero. A good final model should be able to correct for this, but I think its unnecessary to force it to.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.