I have a dataset {$X_i$, $Y_i$}. $Y_i$ - real value targets, they can be negative. The task is to teach classifier $f: X_i \rightarrow \{0, 1\}$ to maximize the sum below $$ L(f) = \sum_i f(X_i) * Y_i $$ So $f(X_i) = I[Y_i > 0]$ would be the ideal classifier. I'm looking for appropriate smooth loss function to teach NN. NN output is probability $p(X_i)$, so $f(X_i) = I[p(X_i) > 0.5]$. I have tried the next loss function: $$ BadLoss(p) = \sum_i p(X_i) * Y_i $$ but because $Y_i$ can be negative it is bad-defined loss and NN converges to nothing good. The best results I got were with usual logloss: $$ Loss(p) = \sum_i logloss(p(X_i), \; I[Y_i > 0]) $$

Can be there better loss functions to maximize the sum $L(f)$?

  • $\begingroup$ You say your targets are real, yet you want to classify them? If you have two classes use the traditional $y \in {0,1}$ assignments with the cross-entropy loss. $\endgroup$ – Emre Jan 24 '17 at 5:36
  • $\begingroup$ @Emre, I don't want to classify them. Classifier for signs of $Y_i$ is just a hack that works to maximize $L(f)$. Is there a better way? I want to optimize loss that more similar to L(f). $\endgroup$ – alex_io Jan 24 '17 at 5:44

Define $Z_i = 1[Y_i>0]$, i.e., $Z_i = 1$ if $Y_i > 0$, else $Z_i = -1$. Now the data set $(X_i,Z_i)$ defines a two-class boolean classification problem. The solution to that boolean classification problem with highest accuracy solves your original problem (minimizes your loss function).

So, you are basically asking how to train a neural network to solve a two-class boolean classification problem. A reasonable approach is to put a softmax layer at the output of the neural network and train it using the cross-entropy loss, as usual. During test time, to pick a class, you pick whichever of the two classes has the higher likelihood output from the softmax stage. In other words, at training time, you look at the actual outputs (continuous-valued) from the softmax layer and use the cross-entropy; at test time, you compare the two outputs and pick whichever is larger to choose the classification (and apart from that ignore their exact values).

The shortcoming of this approach is that it doesn't take into account the value of the $Y_i$'s, only their sign. To improve this, use weights on the samples. We're again going to train a boolean classifier on the training set $(X_i,Z_i)$, but this time we'll weight each sample in the training set differently. Put the weight $Y_i$ on the sample $(X_i,Z_i)$ in your training set. In other words, when training the boolean classifier, the loss function will be a weighted sum of the loss for each sample:

$$\text{Loss}(\theta) = \sum_i Y_i \cdot \ell(f_\theta(X_i),Z_i)$$

where $\theta$ are the model parameters, $f_\theta(X_i)$ is the output of the classifier on input $X_i$, and $\ell(\cdot,\cdot)$ is the cross-entropy loss. Notice that errors in samples where $Y_i$ is large are penalized more, and errors in samples where $Y_i$ is small are penalized less; this is appropriate, as that's exactly what will happen when you use the classifier $f_\theta$ to compute $L(f_\theta)$. This should improve on the approach you're taking.

| improve this answer | |
  • $\begingroup$ Thanks for the answer. But that is exactly what I did. It worked pretty well. But maybe there is a way to take into account values of $Y_i$ not only signs. $\endgroup$ – alex_io Jan 24 '17 at 8:09
  • $\begingroup$ @alex_io, ahh, I see! OK, see edited answer, where I describe an improvement, using weights. $\endgroup$ – D.W. Jan 24 '17 at 18:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.