I am building a Neural Network for a binary classification problem where the Bayes error (lowest possible error rate) is probably close to 50%.

What makes the task easier is that I don't need to make a prediction for each observation of the test sample. I only want to make a prediction for the observations where the model has a fairly high confidence. However a high rate at which  predictions are made is better than a low one.

So far, I have used a standard neural network (feed-forward, cross-entropy loss, L2 regularization and sigmoid activation on final node). In the testing sample, I only take into account the observations for which the final node's value $(\hat{Y}_i)$ is outside of an interval of low confidence:
$$\text{predicted class}_i = \begin{cases}
 1 &\text{ if } \hat{Y}_i > 0.5 + a \\ 
 0 &\text{ if } \hat{Y}_i < 0.5 - a \\
 \text{NA} &\text{else}
\end{cases} \\
\text{where } a\in [0, 0.5] \text{ indicates the level of confidence required}$$

To tune the hyperparameters (including $a$), I have designed a metric that depends positively on: 

 - Test-sample accuracy (only counting predictions different from NA)
 - Percentage of predictions that are different from NA. 

I am not yet satisfied with the performance achieved with this approach, and I am sure that there are smarter ways to approach this, for example a custom loss function. Advices, links to articles, or even related search keywords are welcome.