I have a loss function that minimizes the error according to what I want the neural network to do. The problem is, that it is a nondifferentiable function. How can I handle this?

the loss function: $(1-y) \cdot log(1-p) + min((1-y)-(y \cdot log(p)))$

  • $y$: target
  • $p$: prediction
  • len((1-y)-(y*log(p))) = len(y) = len(p)

I have tried to smooth the minimum, but I am not sure this is good enough. As you can see, the min operator is nondifferentiable

How to handle a nondifferentiable loss function with Neural Networks?

  • 6
    $\begingroup$ min needs at least two variables but you only have one, i.e min(x,y) or min(x,y,z)? $\endgroup$ – serali Jan 13 '20 at 17:39
  • $\begingroup$ Can you describe what y and p are in this context? $\endgroup$ – zachdj Jan 13 '20 at 17:56
  • 1
    $\begingroup$ y is the target, and p is the prediction @zachdj $\endgroup$ – user2974655 Jan 13 '20 at 19:19

You can optimize with non-gradient based methods. The field is called derivative-free optimization.

Local Search is one common approach for derivative-free optimization of neural networks.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.