# Combining multiple neural networks with different activation functions

I have 3 neural networks where each has as a different activation function: Sigmoid, Tanh and Softmax. I am planning to average their final predictions, but as we know the functions doesn't have the same range values.

P = (P1 + P2 + P3)/3


Where 0 < P1 < 1, -1 < P2 < 1, 0 < P3 < 1

Can I directly average the predictions or I need to perform a normalization to have all prediction fall into the same interval ?

• I imagine you would have to normalize the ranges. However, before doing that I would ask myself whether it is correct to average the predictions and whether that average would be more meaningful than not. Oct 10, 2019 at 19:58
• I am trying to create neural networks and have a voting system which reflects the ensemble prediction. I read a lot about averaging weights when using ensemble; but I don't understand what you mean Oct 10, 2019 at 20:01
• If you're creating an ensemble, why do the final-layer activations differ? Shouldn't all models in the ensemble be trying to solve the same problem? Oct 10, 2019 at 20:12
• Well I am only using neural networks, if I keep the same parameters for each model, than average the predictions wouldn't make any sense as mention by @Jason. But I am not only trying to change the final activation function, but many stuff like cost, number of hidden layers, etc. Oct 10, 2019 at 20:14

## 1 Answer

As you are trying to average out the values, and given the three have different domains, it makes sense to bring all in same domain before averaging out. You can normalize P2 (tanh) to 0-1 and then average the values.

If you want to try another way, you can combine these three networks and take the three outputs p1, p2, and p3 and input it to some more dense layers and finally predict the single output. This way instead of averaging out, you are bringing in some nonlinearity which will be helpful to learn the task.