Binary classification and numerical labels

Question

I am trying to create a sentiment analysis model using a dataset that have ~50000 positive tweets that i labeled as 1, ~50000 negative tweets that i have labeled as 0. Also i have acquired ~10000 tweets that are neutral.

Due to the low number of neutral tweets my thinking is to label neutral with 0.5 and train the model using binary crossentropy as loss function. My output layer is 1 neuron with sigmoid activation function so prediction value would be between (0,1) .

Is my thinking right or it will mess the accuracy?

Answer 1

If you’re going to have more than two labels, you need to go with a softmax activation and a loss for multi class classification, ie cross entropy loss.

Also, be cautious for multi-class versus multi-label (below).

---

Multi-class

One-of-many classification. Each sample can belong to ONE of $C$ classes. The model will have $C$ output neurons that can be gathered in a vector $s$ (Scores). The target (ground truth) vector $t$ will be a one-hot vector with a positive class and $C−1$ negative classes. This task is treated as a single classification problem of samples in one of $C$ classes.

Multi-label

Each sample can belong to more than one class. The model will have as well $C$ output neurons. The target vector $t$ can have more than a positive class, so it will be a vector of 0s and 1s with $C$ dimensionality. This task is treated as $C$ different binary $(C′=2,t′=0 \ or \ t′=1)$ and independent classification problems, where each output neuron decides if a sample belongs to a class or not.

source: https://gombru.github.io/2018/05/23/cross_entropy_loss/

Answer 2

Binary cross-entropy is only a suitable loss function if you are performing binary (two-class) classification problems. If you add a third "neutral" class, it's no longer appropriate. There are two ways you could frame your problem:

1. Multi-class classification. In that case, the suggestion to use an output layer with three neurons and softmax activation to normalise the outputs is most appropriate. In Keras, for example, the appropriate losses would be CategoricalCrossentropy (or the sparse equivalent).

2. Regression. You could alternatively train the network to output a "positivity score" between 0 and 1, where 0.5 represents neutral. The output layer in this case could just be a single neuron with sigmoidal activation, which would bound the output to the interval $(0, 1)$. You could use a loss such as mean square error and a suitable regression metric to monitor whether your model is working. This is reasonable because you can think of the positivity of some text as a continuous quantity rather than discrete categories.

Try each approach and see which seems to work better!

Answer 3

In multiclass problem use softmax activation function. For example, in Keras you put 3 neurons:

model.add(Dense(3, activation='softmax'))

As a loss function you can choose Categorical Crossentropy:

loss=tf.keras.losses.CategoricalCrossentropy()

So that you compile model:

model.compile(loss=tf.keras.losses.CategoricalCrossentropy(), 
              optimizer=tf.keras.optimizers.Adamax(learning_rate=0.01), 
              metrics=['acc']
             )

Don't forget also to add class weights due to class imbalance 50000:50000:10000 for Positive:Negative:Neutral classes. Otherwise, accuracy metric makes no sense if you are care for example about neutral class more.