# best activation function for ensemble?

i have created some logistic regression model (different preprocessing) with softmax function. and i mix all model with an ensemble with a hierarchical method. so the output of all model (base) will be used as input for the final model (logistic regression too).

the default base model used a softmax function. i think transform a confident value into a probability will lose much information. so i have a plan to change the softmax into activation function.

what i learn in my CNN class, Relu is the best default activation function for an image. but my case is multi-class classification email.

which activation function should i choose? Relu, sigmoid or other?

thanks

• LeakyRelu, Relu6 or rather you can create your own Relu version with adaptivity on the -ve axis – Aditya Oct 2 '18 at 5:55

I think you will have to experiment; there isn't generally a one-activation-fits-all for hierarchical models. Give the ones you mentioned a try, but perhaps starting with sigmoid and tanh and also a LeakyReLU.
The reason I'd perhaps leave out a normal ReLU at first is because that too would potential trim out information before the final model, as it simply removes negative weights to zero. A good final model should be able to correct for this, but I think its unnecessary to force it to.