Training deep neural networks with ReLU output layer for verification

Question

Most algorithms for verification of deep neural network require ReLU activation functions in each layer (e.g. Reluplex).
I have a binary classification task with classes 0 and 1. The main problem I see is that ReLU is not bound to [0, 1] like the Sigmoid or Softmax activation function.

How can one use/train a DNN with ReLU at the last layer for binary classification tasks with one output unit? What loss function could work?
Or do I have to reformulate my problem from classification to regression?

Answer 1

For binary classification tasks, you do not require to use ReLU at the prediction node as you have said, the output from the ReLU activation doesn't match to your need. You may want to consider ReLU, Sigmoid, Tanh for the hidden layer(s) and Sigmoid or Softmax for the output node for binary classification.

If you study the modern deep learning model architectures, you'll find ReLU often in the hidden layers but not in the prediction layer. Further, you may find this comprehensive blog useful.