What are the best activation functions for Binary text classification in neural networks?

Question

I know that there are many activation functions like Relu, sigmoid, tanh ..etc, I just want to know the best one for my case - Binary text classification.

I have heard that Relu is best for Binary classification (not sure if im correct)

I have used keras to train a model, which is 2 layer , Dense 512, dropout 0.3, activation = "Relu" for these layers,

But the predictions are not upto the mark.

I have also changed the Dense units to 1024, keeping others same, but still I got bad predictions. (Validation accuracy 50%)

So, can i use other activations, or change my model layers (add few more layers) ??

What can be the best option ?

Answer 1

Sigmoid activation produces an output between 0,1 making it suited for binary classification. As far as hyper parameter tuning goes, experimenting is always necessary.

Answer 2

Relu often works well, but ultimately you need to try. If you really have only one hidden layer, you might have a model which cannot properly learn.

I guess you use a sequential, dense model architecture? Try to add more hidden layers. Don‘t add too much capacity for a start. You could try something like 256, 8, 8, 1 with relu.

Dropout is good to prevent overfitting. It is not important for learning per se.

Just a remark for future posts: always include your basic model code if possible. So people can understand what you are really up to. Also a little data description is useful. Especially with text, preprocessing is really important. So the way you treated your data before it goes into the model is relevant.

Answer 3

As you are not specific about how to convert text input into numbers to feed to the model, I can only answer your question in general.

Firstly, for the last layer of binary classification, the activation function is normally softmax (if you define the last layer with 2 nodes) or sigmoid (if the last layer has 1 node).

For other layers, it is hard to tell sigmoid or relu is better. But in my experience, relu works better with more complicated models.