Question
What is the loss function for each different activation function?
Background
The choice of the loss function of a neural network depends on the activation function. For sigmoid activation, cross entropy log loss results in simple gradient form for weight update z(z - label) * x
where z is the output of the neuron.
This simplicity with the log loss is possible because the derivative of sigmoid make it possible, in my understanding. The activation function other than sigmoid which does not have this nature of sigmoid would not be a good combination with the log loss. Then what are the loss function for ReLu, ELU, SELU?