The mean square loss function is the standard for regression neural networks. However, if I have a neural network learning two tasks (two outputs) at once, is it more advisable to train on the sum of the relative errors for the different outputs or the sum of the mean square errors of both tasks?

Intuitively, the mean square loss function favours the task with larger expected values so it's "unfair" to the other task. I'm leaning towards using relative error but are there any caveats to using it as the loss function?

Update: Problem with using relative error is that it's a percentage and therefore the gradient updates will be very small. I think it may be more practical to use an error function proportional to the mean relative errors but scaled so that they are large enough.

  • $\begingroup$ We generally can have a softmax at the end in that case and the activation functions can be set to ReLU/LeakyReLu etc.. $\endgroup$
    – Aditya
    Apr 5 '18 at 2:10
  • 1
    $\begingroup$ @Aditya: Isn't softmax usually used in classifiers? I'm looking to output actual values for my predictions. $\endgroup$ Apr 5 '18 at 2:44
  • $\begingroup$ Softmax can work for regression, you can also try logit or linear actuvation. $\endgroup$
    – JahKnows
    Apr 12 '18 at 5:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.