1
$\begingroup$

I am building a deep neural network based binary classifier, with single output. The loss function I actually want to minimize is $$ \mathcal L(\hat y,y) = \begin{cases} 0, & \text{if $\hat y$ = 0} \\ 1, & \text{if $\hat y$ = 1 & $y$ = 0} \\ \gamma \approx -0.7 , & \text{if $\hat y$ = 1 & $y$ = 1} \end{cases} $$ where $y \in \{0;1\}$ is sample's label, $\hat y \in \{0;1\}$ - classifier's output and $\gamma$ - a hyperparameter. I think this is a case of assymmetric loss. (One can see it as betting: no reward when not betting, stake 1\$, payout 1.7\$ if bet wins)

From what I know by now, this loss function is probably not suited well for backpropagation and gradient descent. Question: Is there a better-suited formulation?

The often used cross-entropy loss doesn't allow for trade-off tuning between precision and recall. LINEX and LINLIN are assymetric by design purpose, but I coundn't find an example of a deep NN trained with them. An alternative could be leaving the loss function as it is and resorting to SPSA, but I'd like to keep it simple if possible.

Edit: I came up with $$ \mathcal L(\hat y,y) = - (\gamma \hat y)^{y}(-\hat y)^{1-y}$$ At the moment, I have no clue if it is going to work for NN learning. I'm concerned (maybe unnecessarily) that, without logarithm, it is not convex regarding NN weights. (Last layer has sigmoid activation.) Here is the log loss shown for comparison. $$ \mathcal L(\hat y,y) = - (y\log \hat y + (1-y)\log(1-\hat y)) = -\log[\hat y^y (1-\hat y)^{1-y}]$$

$\endgroup$
1
$\begingroup$

Rather than 0 use 0.1 or some other small non-zero value. Use the true y as an indicator function, multiply your class probability by your reward component, and sum terms for all possibilities to get a usable loss, like this:

$Loss(y, y_p) = (1-y)(1)(y_p) + (y)(-0.7)(y_p) + (0.1)(1-y_p)$

Where $y_p =P(\hat{y}=1|X)$

| improve this answer | |
$\endgroup$
  • $\begingroup$ thanks for an idea. Should I be concerned by the fact that e.g. for yp=0 the loss is not exactly as in the application? It also seems to be not convex in regard to NN weights (at least if the final layer uses sigmoid activation function) and thus not well suited for gradient descent. $\endgroup$ – Xpector Feb 11 '18 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.