10
$\begingroup$

It seems standard in many neural network packages to pair up the objective function to be minimised with the activation function in the output layer.

For instance, for a linear output layer used for regression it is standard (and often only choice) to have a squared error objective function. Another usual pairing is logistic output and log loss (or cross-entropy). And yet another is softmax and multi log loss.

Using notation, $z$ for pre-activation value (sum of weights times activations from previous layer), $a$ for activation, $y$ for ground truth used for training, $i$ for index of output neuron.

  • Linear activation $a_i=z_i$ goes with squared error $\frac{1}{2} \sum\limits_{\forall i} (y_i-a_i)^2$

  • Sigmoid activation $a_i = \frac{1}{1+e^{-z_i}}$ goes with logloss/cross-entropy objective $-\sum\limits_{\forall i} (y_i*log(a_i) + (1-y_i)*log(1-a_i))$

  • Softmax activation $a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$ goes with multiclass logloss objective $-\sum\limits_{\forall i} (y_i*log(a_i))$

Those are the ones I know, and I expect there are many that I still haven't heard of.

It seems that log loss would only work and be numerically stable when the output and targets are in range [0,1]. So it may not make sense to try linear output layer with a logloss objective function. Unless there is a more general logloss function that can cope with values of $y$ that are outside of the range?

However, it doesn't seem quite so bad to try sigmoid output with a squared error objective. It should be stable and converge at least.

I understand that some of the design behind these pairings is that it makes the formula for $\frac{\delta E}{\delta z}$ - where $E$ is the value of the objective function - easy for back propagation. But it should still be possible to find that derivative using other pairings. Also, there are many other activation functions that are not commonly seen in output layers, but feasibly could be, such as tanh, and where it is not clear what objective function could be applied.

Are there any situations when designing the architecture of a neural network, that you would or should use "non-standard" pairings of output activation and objective functions?

$\endgroup$
4
$\begingroup$

It is not so much which activation function which you use which determines which loss funtion you should use, but rather what the interpretation you have of the output is.

If the output is supposed to be a probability, then log-loss is the way to go.

If the output is a generic value then mean squared error is default way to go. So for example, if your output was a grey scale pixel with grey-scale labelled by a number from 0 to 1, it might make sense to use a sigmoid activation function with a mean squared error objective function.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.