When used in a linear model, a convex loss function guarantees a unique global minimum for the parameters, which can be found by local optimization methods.
However, when the model is nonlinear (e.g. MLPs), local minima are possible for a convex loss.
Are there any benefits to a convex loss function when the model is nonlinear? Can convexity be completely disregarded in the nonlinear case?
Another benefit of a convex loss function is it will have faster convergence for all models, both linear and nonlinear. There will be even faster convergence in a convex loss function if a momentum term is added to gradient descent.
However often in real-world scenarios and with many models types, the loss function is not guaranteed to be convex. It is not clear what "convexity be completely disregarded" means. Machine learning systems should be designed to be robust to non-convex loss functions in order to find useful parameters in a board range of problems.
