1
$\begingroup$

I have 3 neural networks where each has as a different activation function: Sigmoid, Tanh and Softmax. I am planning to average their final predictions, but as we know the functions doesn't have the same range values. 

P = (P1 + P2 + P3)/3

Where 0 < P1 < 1, -1 < P2 < 1, 0 < P3 < 1

Can I directly average the predictions or I need to perform a normalization to have all prediction fall into the same interval ?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ I imagine you would have to normalize the ranges. However, before doing that I would ask myself whether it is correct to average the predictions and whether that average would be more meaningful than not. $\endgroup$
    – Jason K Lai
    Commented Oct 10, 2019 at 19:58
  • $\begingroup$ I am trying to create neural networks and have a voting system which reflects the ensemble prediction. I read a lot about averaging weights when using ensemble; but I don't understand what you mean $\endgroup$
    – U. User
    Commented Oct 10, 2019 at 20:01
  • 1
    $\begingroup$ If you're creating an ensemble, why do the final-layer activations differ? Shouldn't all models in the ensemble be trying to solve the same problem? $\endgroup$
    – zachdj
    Commented Oct 10, 2019 at 20:12
  • $\begingroup$ Well I am only using neural networks, if I keep the same parameters for each model, than average the predictions wouldn't make any sense as mention by @Jason. But I am not only trying to change the final activation function, but many stuff like cost, number of hidden layers, etc. $\endgroup$
    – U. User
    Commented Oct 10, 2019 at 20:14

1 Answer 1

Reset to default
1
$\begingroup$

As you are trying to average out the values, and given the three have different domains, it makes sense to bring all in same domain before averaging out. You can normalize P2 (tanh) to 0-1 and then average the values.

If you want to try another way, you can combine these three networks and take the three outputs p1, p2, and p3 and input it to some more dense layers and finally predict the single output. This way instead of averaging out, you are bringing in some nonlinearity which will be helpful to learn the task.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.