I have a loss function that minimizes the error according to what I want the neural network to do. The problem is, that it is a nondifferentiable function. How can I handle this?

the loss function: $(1-y) \cdot log(1-p) + min((1-y)-(y \cdot log(p)))$

  • $y$: target
  • $p$: prediction
  • len((1-y)-(y*log(p))) = len(y) = len(p)

I have tried to smooth the minimum, but I am not sure this is good enough. As you can see, the min operator is nondifferentiable

How to handle a nondifferentiable loss function with Neural Networks?

  • 6
    $\begingroup$ min needs at least two variables but you only have one, i.e min(x,y) or min(x,y,z)? $\endgroup$
    – serali
    Jan 13, 2020 at 17:39
  • $\begingroup$ Can you describe what y and p are in this context? $\endgroup$
    – zachdj
    Jan 13, 2020 at 17:56
  • 1
    $\begingroup$ y is the target, and p is the prediction @zachdj $\endgroup$ Jan 13, 2020 at 19:19

1 Answer 1


You can optimize with non-gradient based methods. The field is called derivative-free optimization.

Local Search is one common approach for derivative-free optimization of neural networks.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.