5
$\begingroup$

There are different types of cost functions like Cross-entropy, absolute error, mean squared error.

When can we expect the cost function to be non-convex? Does this depend upon the type of cost function we choose? Or does this depend upon the Model we choose like when we have higher order polynomial Model?

$\endgroup$
2
  • 1
    $\begingroup$ I think it is not convex in truest sense but we try to make it so and also make sure not to get trapped in a local minima $\endgroup$
    – DuttaA
    Jun 26, 2018 at 12:21
  • 1
    $\begingroup$ It all depends on your final function which is going to be optimised. $\endgroup$ Aug 25, 2018 at 17:29

2 Answers 2

1
$\begingroup$

The cost function is convex if its Second Order Derivative is positive semidefinite (i.e. $\geq0$ ).

But this definition depends on the function with respect to which you take the derivative. This convexity changes when we are talking about Neural Networks, as in that case, our derivatives are taken with respect to the weights.

Refer: Why Cross Entropy is convex, but not necessarily in neural nets and Why RMSE is convex but not necessarily in neural nets

$\endgroup$
1
$\begingroup$

$f(x)$ is convex when $f(a)<f(x)<f(b)$ for every $a<x<b$. Overall, a function with a positive second derivative is convex.

The MSE objective is of the form

$MSE = ∑(y_{true} − 𝑦_{pred})^2$

The second derivative is positive. So, MSE is convex. You can follow this procedure for the rest of the functions.

Here is also another answer to this question.

$\endgroup$
1
  • $\begingroup$ One small correction, it should be SSE (sum of squared errors) and not MSE (mean squared error). MSE = SSE/N. $\endgroup$
    – Aryan
    Mar 7, 2022 at 2:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.