1
$\begingroup$

When used in a linear model, a convex loss function guarantees a unique global minimum for the parameters, which can be found by local optimization methods.

However, when the model is nonlinear (e.g. MLPs), local minima are possible for a convex loss.

Are there any benefits to a convex loss function when the model is nonlinear? Can convexity be completely disregarded in the nonlinear case?

$\endgroup$
1
$\begingroup$

Another benefit of a convex loss function is it will have faster convergence for all models, both linear and nonlinear. There will be even faster convergence in a convex loss function if a momentum term is added to gradient descent.

However often in real-world scenarios and with many models types, the loss function is not guaranteed to be convex. It is not clear what "convexity be completely disregarded" means. Machine learning systems should be designed to be robust to non-convex loss functions in order to find useful parameters in a board range of problems.

$\endgroup$
1
  • $\begingroup$ Thanks for the answer. What I meant by "disregarded" is whether the convexity of loss has any benefit at all in the multiple optima case (i.e. nonlinear model). Your first paragraph shows that speed of convergence is one such benefit. Are there any others that you know of? $\endgroup$
    – user1337
    Sep 13 at 5:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.