1
$\begingroup$

I am going to do Sentiment Analysis over some tweet texts. So, in summary we have three classes: Positive, Neutral, Negative. If I apply Softmax in the last layer, I will have the probability for each class for each piece of text. we know that in Softmax:

P(pos) + P(neu) + P(neg) = 1

My question: suppose that we have a piece of text in Positive label. So, do we have to have these probabilities in this order:

P(pos) > P(neu) > P(neg)

What does it mean when we have them in this order:

P(pos) > P(neg) > P(neu)

Can we conclude anything from this? For example, can we say with confidence that the label is Positive like as before?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

If you have a text in positive label, and your model think it is positive then the positive probability your model output will be the largest.

If you ask your model which is the second most likely label that you(your model) think this text sample belong to, your model's answer is the class that has the second largest probability in the output, and so on.

In summary, your model rank the class from most likely to less likely to your sample. So the order of the probabilities depend on your model belief, such that the most likely class will has the largest probability and the least likely class will has the least probability.

My question: suppose that we have a piece of text in Positive label. So, do we have to have these probabilities in this order: P(pos) > P(neu) > P(neg)

Not exactly, it depends on your model belief, which depends on how good is your data to express the idea of positive, neutral and negative. But usually when use logistic regression to classify 3 class positive, neutral and negative, people will set a threshold for positive, neutral and negative in the probability range, for example: > 0.7 is positive, in [0.4, 0.7] is neutral and the remaining is negative. By doing this, we implicitly assume that the probabilities are indeed have order as you said. This is because we assume that there is an order between positive, neutral and negative, such that neutral is between positive and negative. But if we are dealing with another problem for example classify dog, cat and fist, then I don't think we can assume the order.

What does it mean when we have them in this order: P(pos) > P(neg) > P(neu)

It means that the model believe the most likely class to your sample is positive, the second most likely is negative, and the least likely is neutral.

Can we conclude anything from this? For example, can we say with confidence that the label is Positive like as before?

In my opinion, the model is confident with its answer, if we choose to believe it, then we can confidently say that the sample's class is positive as before.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I was going to use BERT + Classifier to do opinion mining and find out which tweet is positive or negative about a specific topic. we have about 60,000 tweets for training which I do not know is enough or not. There are remarkable tweets that identified as NEG but they are POS which are not expected. I am going to find out this. Thanks $\endgroup$
    – Mahdi Amrollahi
    Nov 28, 2021 at 1:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.