3
$\begingroup$

I have a loss function that minimizes the error according to what I want the neural network to do. The problem is, that it is a nondifferentiable function. How can I handle this?

the loss function: $(1-y) \cdot log(1-p) + min((1-y)-(y \cdot log(p)))$

  • $y$: target
  • $p$: prediction
  • len((1-y)-(y*log(p))) = len(y) = len(p)

I have tried to smooth the minimum, but I am not sure this is good enough. As you can see, the min operator is nondifferentiable

How to handle a nondifferentiable loss function with Neural Networks?

Improve this question
$\endgroup$
3
  • 6
    $\begingroup$ min needs at least two variables but you only have one, i.e min(x,y) or min(x,y,z)? $\endgroup$
    – serali
    Jan 13, 2020 at 17:39
  • $\begingroup$ Can you describe what y and p are in this context? $\endgroup$
    – zachdj
    Jan 13, 2020 at 17:56
  • 1
    $\begingroup$ y is the target, and p is the prediction @zachdj $\endgroup$
    – user2974655
    Jan 13, 2020 at 19:19

1 Answer 1

Reset to default
2
$\begingroup$

You can optimize with non-gradient based methods. The field is called derivative-free optimization.

Local Search is one common approach for derivative-free optimization of neural networks.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.