I am building a Neural Network for a binary classification problem where the Bayes error (lowest possible error rate) is probably close to 50%. What makes the task easier is that I don't need to make a prediction for each observation of the test sample. I only want to make a prediction for the observations where the model has a fairly high confidence. However a high rate at which predictions are made is better than a low one. So far, I have used a standard neural network (feed-forward, cross-entropy loss, L2 regularization and sigmoid activation on final node). In the testing sample, I only take into account the observations for which the final node's value $(\hat{Y}_i)$ is outside of an interval of low confidence: $$\text{predicted class}_i = \begin{cases} 1 &\text{ if } \hat{Y}_i > 0.5 + a \\ 0 &\text{ if } \hat{Y}_i < 0.5 - a \\ \text{NA} &\text{else} \end{cases} \\ \text{where } a\in [0, 0.5] \text{ indicates the level of confidence required}$$ To tune the hyperparameters (including $a$), I have designed a metric that depends positively on: - Test-sample accuracy (only counting predictions different from NA) - Percentage of predictions that are different from NA. I am not yet satisfied with the performance achieved with this approach, and I am sure that there are smarter ways to approach this, for example a custom loss function. Advices, links to articles, or even related search keywords are welcome.