I am training a neural network for a multilabel classification problem, so my last layer consist of
n_classes sigmoid neurons.
Now, I know that it is impossible to nail the learning task both because of noise and lack of predictable power in my features. However, I am assuming that the predictable power should be enough to make good predictions at least for some data points. Even if these cases are just a few, I would be happy if the neural network gets at least those with certainty (high precision) and misses the grand majority because of their difficulty (low recall).
Of course, I could train the model normally and then set that threshold that matches my goals, but this doesn't convince me. I would like to introduce this goal in the training process. If some points are hard to classify I want my neural network to discover that during training, forget about them and focus on the other more doable cases.
So far my only idea is to make training robust towards outliers via label smoothing.
Any other ideas to influence the model during training like this?