4
$\begingroup$

I am trying to create a sentiment analysis model using a dataset that have ~50000 positive tweets that i labeled as 1, ~50000 negative tweets that i have labeled as 0. Also i have acquired ~10000 tweets that are neutral.

Due to the low number of neutral tweets my thinking is to label neutral with 0.5 and train the model using binary crossentropy as loss function. My output layer is 1 neuron with sigmoid activation function so prediction value would be between (0,1) .

Is my thinking right or it will mess the accuracy?

$\endgroup$
0
5
$\begingroup$

If you’re going to have more than two labels, you need to go with a softmax activation and a loss for multi class classification, ie cross entropy loss.

Also, be cautious for multi-class versus multi-label (below).


Multi-class

One-of-many classification. Each sample can belong to ONE of $C$ classes. The model will have $C$ output neurons that can be gathered in a vector $s$ (Scores). The target (ground truth) vector $t$ will be a one-hot vector with a positive class and $C−1$ negative classes. This task is treated as a single classification problem of samples in one of $C$ classes.

Multi-label

Each sample can belong to more than one class. The model will have as well $C$ output neurons. The target vector $t$ can have more than a positive class, so it will be a vector of 0s and 1s with $C$ dimensionality. This task is treated as $C$ different binary $(C′=2,t′=0 \ or \ t′=1)$ and independent classification problems, where each output neuron decides if a sample belongs to a class or not.

source: https://gombru.github.io/2018/05/23/cross_entropy_loss/

$\endgroup$
1
  • $\begingroup$ I am sorry i forgot to mention.My output layer is 1 neuron with sigmoid activation function so my output would be between (0,1) . $\endgroup$
    – Antonis
    Feb 14 at 12:03
2
$\begingroup$

Binary cross-entropy is only a suitable loss function if you are performing binary (two-class) classification problems. If you add a third "neutral" class, it's no longer appropriate. There are two ways you could frame your problem:

  1. Multi-class classification. In that case, the suggestion to use an output layer with three neurons and softmax activation to normalise the outputs is most appropriate. In Keras, for example, the appropriate losses would be CategoricalCrossentropy (or the sparse equivalent).

  2. Regression. You could alternatively train the network to output a "positivity score" between 0 and 1, where 0.5 represents neutral. The output layer in this case could just be a single neuron with sigmoidal activation, which would bound the output to the interval $(0, 1)$. You could use a loss such as mean square error and a suitable regression metric to monitor whether your model is working. This is reasonable because you can think of the positivity of some text as a continuous quantity rather than discrete categories.

Try each approach and see which seems to work better!

$\endgroup$
3
  • $\begingroup$ Your answer made my realize that maybe i am using binary crossentropy wrong.My output layer is 1 neuron.Should i change that to mse? $\endgroup$
    – Antonis
    Feb 14 at 11:58
  • $\begingroup$ @Antonis Αντώνη, what both answers are trying to tell you is that binary cross entropy (and sigmoid activations) are for binary problems, which your own problem is no longer, since you now have 3 classes. The answer is not MSE (which is for regression problems, and not for classification ones), but categorical cross-entropy - and yes, you will have to change your final layer also (one unit is no longer sufficient). $\endgroup$
    – desertnaut
    Feb 14 at 18:15
  • $\begingroup$ @desertnaut.I am sorry i should be more explanatory.Due to the low number of neutral tweets, I am thinking is better to approach the problem like regression than category. My output will be between (0,1) and I want to add neutral tweets as 0.5, so a neutral sequence could be scored closer to 0.5 than 1 or 0. Sorry for any inconvenience and Thank you for your time. $\endgroup$
    – Antonis
    Feb 15 at 11:17
1
$\begingroup$

In multiclass problem use softmax activation function. For example, in Keras you put 3 neurons:

model.add(Dense(3, activation='softmax'))

As a loss function you can choose Categorical Crossentropy:

loss=tf.keras.losses.CategoricalCrossentropy()

So that you compile model:

model.compile(loss=tf.keras.losses.CategoricalCrossentropy(), 
              optimizer=tf.keras.optimizers.Adamax(learning_rate=0.01), 
              metrics=['acc']
             )

Don't forget also to add class weights due to class imbalance 50000:50000:10000 for Positive:Negative:Neutral classes. Otherwise, accuracy metric makes no sense if you are care for example about neutral class more.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.