1
$\begingroup$

I am training a neural network for a multilabel classification problem, so my last layer consist of n_classes sigmoid neurons.

Now, I know that it is impossible to nail the learning task both because of noise and lack of predictable power in my features. However, I am assuming that the predictable power should be enough to make good predictions at least for some data points. Even if these cases are just a few, I would be happy if the neural network gets at least those with certainty (high precision) and misses the grand majority because of their difficulty (low recall).

Of course, I could train the model normally and then set that threshold that matches my goals, but this doesn't convince me. I would like to introduce this goal in the training process. If some points are hard to classify I want my neural network to discover that during training, forget about them and focus on the other more doable cases.

So far my only idea is to make training robust towards outliers via label smoothing.

Any other ideas to influence the model during training like this?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

I think you have two options that don't exclude each other:

  1. Check precision/recall metrics constantly during training, on Validation data. In this way, during experimentation and hyperparameter tuning, you will choose models that are more favourable for precision rather than recall.

  2. Customize your loss function to correct the model in the right way. The problem is that precision, recall, F1 score, they are all non differentiable functions, i.e. they cannot enter a Loss function as they are. For that reason, some tried to approximate these metrics in a differentiable way. One example is the soft-F1 score, as explained in this article. If you read the function code, you will find differentiable approximations of TP, TN, FP, FN's. You can copy those lines to customize your personalized pseudo-precision Loss function.

Improve this answer
$\endgroup$
4
  • $\begingroup$ How does MAE work as a loss function, then? $\endgroup$
    – Dave
    Commented May 4, 2020 at 2:08
  • $\begingroup$ MAE is differentiable. The model can use it to backpropagate the error through the nodes, and update the trainable parameters. However, MAE is used for regression tasks, while yours is a classification one. Let me know if I'm missing any detail. $\endgroup$
    – Leevo
    Commented May 4, 2020 at 6:24
  • $\begingroup$ MAE is not differentiable at $0$. $\endgroup$
    – Dave
    Commented May 4, 2020 at 9:13
  • $\begingroup$ You're right mathematically, in practice it is made differentiable at 0 by forcing it to zero by all common deep learning libraries. $\endgroup$
    – Leevo
    Commented May 4, 2020 at 19:49
1
$\begingroup$

edited As a first approximation to your goal and without much effort needed, I would try to customize your validation metric function (for instance, based on what you want, a metric which applies different weights on precision and recall) and pass it via the metrics input parameter of your network.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Precision, Recall, F1 scores, are not differentiable functions. They cannot enter as they are into a custom Loss for a Neural Network. Please elaborate on ways on how to overcome this problem, otherwise it's misleading. $\endgroup$
    – Leevo
    Commented May 3, 2020 at 10:50
  • 1
    $\begingroup$ You are completely right, I actually meant the validation metric rather than the loss function $\endgroup$
    – German C M
    Commented May 3, 2020 at 11:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.