1
$\begingroup$

I have a model X that takes a review and predict the position of the aspect and the polarity:

input: Serves really good sushi. (review)
output: sushi > Positive (aspect > polarity).

I've seen that this task in the literature has many different ways to be evaluated. Also, some datasets/models include the sentiment words (in our example, 'really good').

Anyway, given the above input & output, how can I evaluate my model? on aspect words? on polarity? should I validate that the polarity referred to the correct aspect before counting it as a hit?

Improve this question
$\endgroup$
5
  • $\begingroup$ That is also my question, did you get any answer? $\endgroup$
    – Mahdi Amrollahi
    Commented Jan 13, 2022 at 14:29
  • 1
    $\begingroup$ Yes @MahdiAmrollahi , I'll answer it now. $\endgroup$
    – Minions
    Commented Jan 13, 2022 at 16:19
  • $\begingroup$ What exactly mean sushi > pos ? Do you mean, the polarity of that sentence about sushi is positive? Is that it? $\endgroup$
    – Mahdi Amrollahi
    Commented Jan 13, 2022 at 16:47
  • 1
    $\begingroup$ @MahdiAmrollahi the polarity of the aspect sushi is positive. In ABSA, the polarity is per Aspect, not per sentence. If you're working on an ASBA task, I'll be more than happy to help. $\endgroup$
    – Minions
    Commented Jan 13, 2022 at 16:50
  • $\begingroup$ Thanks, I got it. $\endgroup$
    – Mahdi Amrollahi
    Commented Jan 13, 2022 at 17:10

1 Answer 1

Reset to default
2
$\begingroup$

After a long literature review, I was able to answer my question:

There are several tasks, so it depends:

  • Aspect extraction: measure the accuracy (or any other metric, usually F1 score) of detecting sushi
  • Opinion extraction: measure the accuracy of detecting good
  • Sentiment classification: measure the accuracy for correctly classifying sushi as Positive
  • End-to-end ABSA: the accuracy will be measured based on two steps, 1 and 3. If the model detects the aspect and the sentiment correctly, then this is a correct case. If the model missed one of them, the case will be considered wrong. This category is the answer to my above question
  1. I've seen in SemEval shared tasks (2014, 2015, 2016) that they have another task which is category classification of aspects. For instance, considering the example above, a model should give sushi a food label.

Thus, it depends on what do you want to do exactly, or what is your task. I hope I didn't miss anything.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.