# Combining multiple neural networks with different activation functions

I have 3 neural networks where each has as a different activation function: Sigmoid, Tanh and Softmax. I am planning to average their final predictions, but as we know the functions doesn't have the same range values.

P = (P1 + P2 + P3)/3


Where 0 < P1 < 1, -1 < P2 < 1, 0 < P3 < 1

Can I directly average the predictions or I need to perform a normalization to have all prediction fall into the same interval ?

• I imagine you would have to normalize the ranges. However, before doing that I would ask myself whether it is correct to average the predictions and whether that average would be more meaningful than not. Oct 10 '19 at 19:58
• I am trying to create neural networks and have a voting system which reflects the ensemble prediction. I read a lot about averaging weights when using ensemble; but I don't understand what you mean Oct 10 '19 at 20:01
• If you're creating an ensemble, why do the final-layer activations differ? Shouldn't all models in the ensemble be trying to solve the same problem? Oct 10 '19 at 20:12
• Well I am only using neural networks, if I keep the same parameters for each model, than average the predictions wouldn't make any sense as mention by @Jason. But I am not only trying to change the final activation function, but many stuff like cost, number of hidden layers, etc. Oct 10 '19 at 20:14